Control apparatus and control method

ABSTRACT

There is provided a control apparatus that includes an estimating section calculating motion of an entire image on the basis of an image signal corresponding to an optical image of a living organism input from an imaging section of a medical observation apparatus, to estimate blurring of the entire image according to a result of the calculation, and a control section controlling an operation related to correction of the blurring of the entire image by controlling a coefficient for controlling an amount of correction of the blurring on the basis of a zoom magnification of the imaging section such that a degree of correction of the blurring increases consistently with the zoom magnification.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2018/004051 filed on Feb. 6, 2018, which claimspriority benefit of Japanese Patent Application No. JP 2017-051183 filedin the Japan Patent Office on Mar. 16, 2017. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates to a control apparatus, a control method,and a program.

BACKGROUND ART

In recent years, with developed surgical procedures and instruments,surgery (what is called microsurgery) has frequently been performed inwhich various treatments are offered while a diseased site is beingobserved with a medical observation apparatus such as a surgicalmicroscope or an endoscope. Additionally, such medical observationapparatuses are not limited to apparatuses enabling the diseased site tobe optically observed but include apparatuses causing a displayapparatus such as a monitor or a display to display, as an electronicimage, an image of the diseased site captured using an imaging section(camera) or the like.

CITATION LIST Patent Literature

[PTL 1]

-   Japanese Patent Laid-Open No. 2015-139646

SUMMARY Technical Problem

Incidentally, with an endoscope apparatus, in a case where a hand or thelike holding a camera head is shaken, motion of the shake is transmittedto an objective lens, and thus, image blurring may be caused by theshake of the hand or the like. Accordingly, in recent years, anendoscope apparatus has been proposed to which a technique forcorrecting the image blurring caused by the shake of the hand or thelike is applied. For example, PTL 1 discloses an example of thetechnique for correcting the image blurring caused by the shake of thehand or the like in the endoscope apparatus.

On the other hand, observation of a living organism with a medicalobservation apparatus such as an endoscope involves variations inconditions, environments, and the like related to the observation, andthe effect of correction of the image blurring on the visibility of anobservation target (living organism or the like) varies according tocircumstances of the moment. Thus, even in a case where the imageblurring is corrected under the same conditions as those in certaincircumstances, the visibility of the observation target is notnecessarily improved in other circumstances.

Thus, the present disclosure proposes a control apparatus, a controlmethod, and a program that can correct the image blurring in a moresuitable manner according to the circumstances related to theobservation of the living organism.

Solution to Problem

According to the present disclosure, a control apparatus is providedthat includes an estimating section calculating motion of an entireimage on the basis of an image signal corresponding to an optical imageof a living organism input from an imaging section of a medicalobservation apparatus, to estimate blurring of the entire imageaccording to a result of the calculation, and a control sectioncontrolling an operation related to correction of the blurring of theentire image by controlling a coefficient for controlling an amount ofcorrection of the blurring on the basis of a zoom magnification of theimaging section such that a degree of correction of the blurringincreases consistently with the zoom magnification.

Additionally, according to the present disclosure, a control apparatusis provided that includes an estimating section calculating motion of anentire image on the basis of an image signal corresponding to an opticalimage of a living organism input from an imaging section of a medicalobservation apparatus, to estimate blurring of the entire imageaccording to a result of the calculation, and a control sectioncontrolling an operation related to correction of the blurring of theentire image on the basis of a ratio of a region of the living organismin the image signal, the region being shielded by a subject differentfrom the living organism.

Additionally, according to the present disclosure, a control method isprovided that includes calculating, by a computer, motion of an entireimage on the basis of an image signal corresponding to an optical imageof a living organism input from an imaging section of a medicalobservation apparatus, to estimate blurring of the entire imageaccording to a result of the calculation, and controlling, by thecomputer, an operation related to correction of the blurring of theentire image by controlling a coefficient for controlling an amount ofcorrection of the blurring on the basis of a zoom magnification of theimaging section such that a degree of correction of the blurringincreases consistently with the zoom magnification.

Additionally, according to the present disclosure, a control method isprovided that includes calculating, by a computer, motion of an entireimage on the basis of an image signal corresponding to an optical imageof a living organism input from an imaging section of a medicalobservation apparatus, to estimate blurring of the entire imageaccording to a result of the calculation, and controlling, by thecomputer, an operation related to correction of the blurring of theentire image on the basis of a ratio of a region of the living organismin the image signal, the region being shielded by a subject differentfrom the living organism.

Additionally, according to the present disclosure, a program is providedthat causes a computer to execute calculating motion of an entire imageon the basis of an image signal corresponding to an optical image of aliving organism input from an imaging section of a medical observationapparatus, to estimate blurring of the entire image according to aresult of the calculation, and controlling an operation related tocorrection of the blurring of the entire image by controlling acoefficient for controlling an amount of correction of the blurring onthe basis of a zoom magnification of the imaging section such that adegree of correction of the blurring increases consistently with thezoom magnification.

Additionally, according to the present disclosure, a program is providedthat causes a computer to execute calculating motion of an entire imageon the basis of an image signal corresponding to an optical image of aliving organism input from an imaging section of a medical observationapparatus, to estimate blurring of the entire image according to aresult of the calculation, and controlling an operation related tocorrection of the blurring of the entire image on the basis of a ratioof a region of the living organism in the image signal, the region beingshielded by a subject different from the living organism.

Advantageous Effect of Invention

As described above, according to the present disclosure, a controlapparatus, a control method, and a program are provided that can correctimage blurring in a more suitable manner according to circumstancesrelated to observation of a living organism.

Note that the above-described effect is not restrictive and that, inaddition to or instead of the above-described effect, any of effectsdisclosed herein or other effects understood from the specification maybe exerted.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of a general configurationof an endoscopic imaging system according to an embodiment of thepresent disclosure.

FIG. 2 is a block diagram illustrating an example of functionalconfigurations of a camera head and a CCU illustrated in FIG. 1.

FIG. 3 is a block diagram illustrating an example of a functionalconfiguration of a control apparatus in a medical observation systemaccording to the embodiment of the present disclosure.

FIG. 4 is a flowchart illustrating an example of a flow of a sequence ofprocessing executed by the control apparatus in the medical observationsystem according to the embodiment.

FIG. 5 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 1.

FIG. 6 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 1.

FIG. 7 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 2.

FIG. 8 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 2.

FIG. 9 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 3.

FIG. 10 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 3.

FIG. 11 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 4.

FIG. 12 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 4.

FIG. 13 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 5.

FIG. 14 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 5.

FIG. 15 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 5.

FIG. 16 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 6.

FIG. 17 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 7.

FIG. 18 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 7.

FIG. 19 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 8.

FIG. 20 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 8.

FIG. 21 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 9.

FIG. 22 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 9.

FIG. 23 is a descriptive diagram illustrating an example of the controlrelated to the correction of image blurring performed by the controlapparatus according to Example 9.

FIG. 24 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 10.

FIG. 25 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 10.

FIG. 26 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 11.

FIG. 27 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 12.

FIG. 28 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 13.

FIG. 29 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 14.

FIG. 30 is a descriptive diagram illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 15.

FIG. 31 is a descriptive diagram illustrating an applied example of amedical observation system according to the embodiment of the presentdisclosure.

FIG. 32 is a functional block diagram illustrating a configurationexample of a hardware configuration of an information processingapparatus constituting a medical observation system according to theembodiment of the present disclosure.

DESCRIPTION OF EMBODIMENT

A preferred embodiment of the present disclosure will be described belowin detail with reference to the accompanying drawings. Note that, in thespecification and the drawings, components having substantially the samefunctional configurations are denoted by the same reference signs andthat duplicate description of these components is omitted.

Note that the description is in the following order.

1. Example of Configuration of Medical Observation System

2. Technical Features

-   -   2.1 Basic Configuration    -   2.2 Processing    -   2.3 Examples

3. Applied Example

4. Example of Hardware Configuration

5. Conclusion

1. Example of Configuration of Medical Observation System

First, with reference to FIG. 1 and FIG. 2, an example of a generalconfiguration of a medical observation system according to theembodiment of the present disclosure will be described.

For example, FIG. 1 is a diagram illustrating an example of a generalconfiguration of a medical observation system to which a techniqueaccording to the present disclosure may be applied. FIG. 1 illustratesan example in which the medical observation system is configured as whatis called an endoscopic surgery system. FIG. 1 illustrates that anoperator (surgeon) 167 is operating on a patient 171 on a patient bed169 using an endoscopic surgery system 100. As illustrated in FIG. 1,the endoscopic surgery system 100 includes an endoscope 101, othersurgical instruments 117, a support arm apparatus 127 supporting theendoscope 101, and a cart 137 in which various apparatuses forendoscopic surgery are loaded.

Endoscopic surgery involves stabbing, into the abdominal wall, aplurality of tubular opening instruments referred to as trocars 125 a to125 d, instead of incising the abdominal wall for laparotomy. Then, alens barrel 103 of the endoscope 101 and the other surgical instruments117 are inserted into the body cavity of the patient 171 through thetrocars 125 a to 125 d. In the illustrated example, as the othersurgical instrument 117, an insufflation tube 119, an energy treatmentinstrument 121, and forceps 123 are inserted into the body cavity of thepatient 171. Additionally, the energy treatment instrument 121 is atreatment instrument used for incision and exfoliation of tissues,sealing of blood vessels, and the like using a high-frequency current orultrasonic vibration. However, the surgical instruments 117 are onlyexamples, and as the surgical instruments 117, various surgicalinstruments generally used in endoscopic surgeries may be used, forexample, tweezers and a retractor.

An image of an affected site in the body cavity of the patient 171captured using the endoscope 101 is displayed on a display apparatus141. While viewing, in real time, the image of the affected sitedisplayed on the display apparatus 141, the operator 167 offerstreatment such as excision of a diseased site using the energy treatmentinstrument 121 and the forceps 123. Note that, although not illustrated,the insufflation tube 119, the energy treatment instrument 121, and theforceps 123 are supported by the operator 167, an assistant, or the likeduring surgery.

(Support Arm Apparatus)

The support arm apparatus 127 includes an arm section 131 extending froma base section 129. In the illustrated example, the arm section 131includes joint sections 133 a, 133 b, and 133 c and links 135 a and 135b, and driven under the control of an arm control apparatus 145. The armsection 131 supports the endoscope 101 and controls the position andposture of the endoscope 101. This allows the position of the endoscope101 to be stably fixed.

(Endoscope)

The endoscope 101 includes the lens barrel 103 including a region havinga predetermined length from a distal end of the lens barrel 103 andinserted into the body cavity of the patient 171, and a camera head 105connected to a proximal end of the lens barrel 103. In the illustratedexample, the endoscope 101 is illustrated that is configured as what iscalled a hard mirror including a hard lens barrel 103. However, theendoscope 101 may be configured as what is called a soft mirrorincluding a soft lens barrel 103.

The lens barrel 103 includes an opening formed at the distal end of thelens barrel 103 and in which an objective lens is fitted. A light sourceapparatus 143 is connected to the endoscope 101, and light generated bythe light source apparatus 143 is guided to the distal end of the lensbarrel by a light guide extending inside the lens barrel 103. The lightis then irradiated via the objective lens toward an observation target(in other words, an imaging target) in the body cavity of the patient171. Note that the endoscope 101 may be a forward-viewing endoscope, aforward-oblique viewing endoscope, or a side-viewing endoscope.

The camera head 105 is internally provided with an optical system and animaging element, and reflected light (observation light) from theobservation target is concentrated on the imaging element by the opticalsystem. The observation light is photoelectrically converted by theimaging element to generate an electric signal corresponding to theobservation light, that is, an image signal corresponding to theobservation image. The image signal is transmitted to a camera controlunit (CCU) 139 as RAW data. Note that the camera head 105 has a functionto adjust magnification and a focal length by appropriately driving theoptical system of the camera head 105.

Note that the camera head 105 may be provided with a plurality ofimaging elements in order to support stereoscopic viewing (3D display)or the like. In this case, a plurality of relay optical systems areprovided in the lens barrel 103 to guide the observation light to eachof the plurality of imaging elements.

(Various Apparatuses Loaded in Cart)

The CCU 139 includes a CPU (Central Processing Unit), a GPU (GraphicsProcessing Unit), and the like to integrally control operations of theendoscope 101 and the display apparatus 141. Specifically, the CCU 139executes various types of image processing, for example, developmentprocessing (demosaic processing), on an image signal received from thecamera head 105 in order to display an image based on the image signal.The CCU 139 provides, to the display apparatus 141, the image signal onwhich the image processing has been executed. Additionally, the CCU 139transmits a control signal to the camera head 105 to control driving ofthe camera head 105. The control signal may include information relatedto imaging conditions such as the magnification and the focal length.

The display apparatus 141 displays, under the control of the CCU 139, animage based on the image signal and subjected to the image processing bythe CCU 139. In a case where the endoscope 101 supports image capturingat high resolution, for example, 4K (the number of horizontal pixels3840×the number of vertical pixels 2160) or 8K (the number of horizontalpixels 7680×the number of vertical pixels 4320) and/or supports 3Ddisplay, the display apparatus 141 used can correspondingly providehigh-resolution display and/or 3D display. In a case where the endoscope101 supports image capturing at high resolution such as 4K or 8K, use ofthe display apparatus 141 having a size of 55 inches or more provides astronger sense of immersion. Additionally, a plurality of the displayapparatuses 141 varying in resolution and size may be provided accordingto an intended use.

The light source apparatus 143 includes a light source such as an LED(light emitting diode), and supplies irradiation light to the endoscope101 when an image of the affected site is captured.

The arm control apparatus 145 includes a processor such as a CPU, andoperates in accordance with a predetermined program to control drivingof the arm section 131 of the support arm apparatus 127 in accordancewith a predetermined control scheme.

The input apparatus 147 is an input interface for the endoscopic surgerysystem 100. A user can input various types of information andinstructions to the endoscopic surgery system 100 via the inputapparatus 147. For example, the user inputs, via the input apparatus147, various types of information related to the surgery such asphysical information regarding the patient and information regarding anoperative procedure. Additionally, for example, the user inputs, via theinput apparatus 147, an instruction to drive the arm section 131, aninstruction to change the imaging conditions for the endoscope 101 (typeof irradiation light, magnification, focal length, and the like), aninstruction to drive the energy treatment instrument 121, and the like.

The type of the input apparatus 147 is not limited, and the inputapparatus 147 may any of various well-known input apparatuses.Applicable examples of the input apparatus 147 include a mouse, akeyboard, a touch panel, a switch, a foot switch 157, and/or a lever. Ina configuration in which a touch panel is used as the input apparatus147, the touch panel may be provided on a display surface of the displayapparatus 141.

Alternatively, the input apparatus 147 is a device worn by the user,such as an eyeglass-type wearable device or an HMD (Head MountedDisplay), and various inputs are provided according to gestures or theline of sight of the user detected by these devices. Additionally, theinput apparatus 147 includes a camera that can detect motion of theuser, and various inputs are provided according to gestures or the lineof sight of the user detected in a video captured by the camera.Furthermore, the input apparatus 147 includes a microphone capable ofcollecting voice of the user, and various aural inputs are provided viathe microphone. Accordingly, the input apparatus 147 is configured toenable various types of information to be input in a non-contact manner.This particularly enables a user belonging to a clean field (forexample, the operator 167) to operate, in a non-contact manner,equipment belonging to an unclean field. Additionally, the user canoperate the equipment without a need to release a hand of the user froma surgical instrument held by the user, thus improving usability for theuser.

A treatment instrument control apparatus 149 controls driving of theenergy treatment instrument 121 for cauterizing or incising tissues,sealing blood vessels, and the like. An insufflation apparatus 151 feedsa gas into the body cavity of the patient 171 via the insufflation tube119 to inflate the body cavity for the purpose of allowing the endoscope101 to provide an appropriate field of view and providing an appropriateworkspace for the operator. A recorder 153 is an apparatus that canrecord various types of information related to the surgery. A printer155 is an apparatus that can print various types of information relatedto the surgery in any of various formats such as a text, an image, or agraph.

Particularly characteristic configurations of the endoscopic surgerysystem 100 will be described below in further detail.

(Support Arm Apparatus)

The support arm apparatus 127 includes a base section 129 correspondingto a base, and the arm section 131 extending from the base section 129.In the illustrated example, the arm section 131 includes the pluralityof joint sections 133 a, 133 b, and 133 c and the plurality of links 135a and 135 b coupled by the joint section 133 b. However, FIG. 1illustrates the configuration of the arm section 131 in a simplifiedmanner for simplification. In actuality, the following may beappropriately set so as to provide the arm section 131 with a desireddegree of freedom: the shapes, numbers, and arrangements of the jointsections 133 a to 133 c and the links 135 a and 135 b, the directions ofrotation axes of the joint sections 133 a to 133 c, and the like. Forexample, the arm section 131 may suitably be configured to have six ormore degrees of freedom. Thus, the endoscope 101 can be freely movedwithin a movable range of the arm section 131, enabling the lens barrel103 of the endoscope 101 to be inserted into the body cavity of thepatient 171 from a desired direction.

The joint sections 133 a to 133 c are provided with actuators andconfigured to be rotatable around predetermined rotation axes underdriving of the actuators. The driving of the actuators is controlled bythe arm control apparatus 145 to control the rotation angle of each ofthe joint sections 133 a to 133 c, thus controlling driving of the armsection 131. Accordingly, control of the position and posture of theendoscope 101 may be implemented. In this case, the arm controlapparatus 145 can control the driving of the arm section 131 usingvarious well-known control schemes such as force control and positioncontrol.

For example, the operator 167 may provide an appropriate operation inputvia the input apparatus 147 (including the foot switch 157) to cause thearm control apparatus 145 to appropriately control the driving of thearm section 131 in accordance with the operation input, thus controllingthe position and posture of the endoscope 101. The control allows theendoscope 101 at the distal end of the arm section 131 to be moved froman optional position to another optional position and then fixedlysupported at a position resulting from the movement. Note that the armsection 131 may be operated on the basis of what is called a masterslave scheme. In this case, the arm section 131 may be remotely operatedby the user via the input apparatus 147 installed at a distance from theoperating room.

Additionally, in a case where force control is applied, what is calledpower assist control may be performed in which the arm control apparatus145 receives an external force from the user and drives the actuatorsfor the joint sections 133 a to 133 c to smoothly move the arm section131 in conjunction with the external force. Thus, when directlycontacting and moving the arm section 131, the user can move the armsection 131 by a relatively weak force. Accordingly, the endoscope 101can be intuitively moved by easier operation, thus improving usabilityfor the user.

Here, in general, in endoscopic surgeries, the endoscope 101 issupported by a surgeon referred to as a scopist. In contrast, the use ofthe support arm apparatus 127 enables the position of the endoscope 101to be more reliably fixed without any manual operation. Accordingly, animage of the affected site can be stably obtained, enabling the surgeryto be smoothly performed.

Note that the arm control apparatus 145 need not necessarily be providedin the cart 137. Additionally, the arm control apparatus 145 need notnecessarily be a single apparatus. For example, the arm controlapparatus 145 may be provided in each of the joint sections 133 a to 133c of the arm section 131 of the support arm apparatus 127, and aplurality of the arm control apparatuses 145 may cooperate with oneanother in implementing driving control of the arm section 131.

(Light Source Apparatus)

The light source apparatus 143 supplies the endoscope 101 withirradiation light for imaging of the affected site. The light sourceapparatus 143 includes, for example, a white light source including anLED, a laser light source, or a combination of the LED and the laserlight source. In this case, in a case where the white light sourceincludes a combination of an R laser light source, a G laser lightsource, and a B laser light source, an output intensity and an outputtiming of each color (each wavelength) can be accurately controlled, andthus, the light source apparatus 143 can adjust white balance of acaptured image. Additionally, in this case, images corresponding to R,G, and B can be captured in a time-division manner by irradiating anobservation target with laser light beams from the R, G, and B laserlight sources in a time-division manner and controlling driving of theimaging elements of the camera head 105 in synchronism with irradiationtimings. This method allows color images to be obtained with no colorfilter provided in each of the imaging elements.

Additionally, driving of the light source apparatus 143 may becontrolled so as to change the intensities of output light beams atpredetermined time intervals. Driving of the imaging elements of thecamera head 105 is controlled in synchronism with timings to change theintensities of the corresponding light beams to acquire images in atime-division manner and the images are synthesized. Then, what iscalled a high-dynamic-range image can be generated that is free fromblocked up shadows and blown out highlights.

Additionally, the light source apparatus 143 may be configured to becapable of supplying light of a predetermined wavelength band enablingspecial light observation. In the special light observation, forexample, narrow band imaging is performed in which the dependence, onwavelength, of light absorption in the body tissue is utilized tocapture a high-contrast image of a predetermined tissue such as a mucousmembrane surface layer blood vessel by irradiation with light of a bandnarrower than the band of the irradiation light for normal observation(that is, white light). Alternatively, in the special light observation,fluorescence observation may be performed in which an image is obtainedusing fluorescence generated by irradiation with excitation light. Inthe fluorescence observation, for example, the body tissue may beirradiated with excitation light, with fluorescence from the body tissueobserved (autofluorescence observation), or a reagent such asindocyanine green (ICG) may be locally injected into the body tissue,which is irradiated with excitation light corresponding to thefluorescent wavelength of the reagent to obtain a fluorescent image. Thelight source apparatus 143 may be configured to be capable of supplyingnarrow band light and/or excitation light enabling such special lightobservation.

(Camera Head and CCU)

With reference to FIG. 2, functions of the camera head 105 of theendoscope 101 and the CCU 139 will be described in further detail. FIG.2 is a block diagram illustrating an example of functionalconfigurations of the camera head 105 and the CCU 139 illustrated inFIG. 1.

With reference to FIG. 2, the camera head 105 includes, as functions ofthe camera head 105, a lens unit 107, an imaging section 109, a drivingsection 111, a communication section 113, and a camera head controlsection 115. Additionally, the CCU 139 includes, as functions of the CCU139, a communication section 159, an image processing section 161, and acontrol section 163. The camera head 105 and the CCU 139 are connectedtogether with a transmission cable 165 so as to be capable ofbidirectional communication.

First, the functional configuration of the camera head 105 will bedescribed. The lens unit 107 is an optical system provided at aconnection with the lens barrel 103. Observation light captured from thedistal end of the lens barrel 103 is guided to the camera head 105 andenters the lens unit 107. The lens unit 107 includes a combination of aplurality of lenses including a zoom lens and a focus lens. The lensunit 107 has optical properties adjusted to concentrate observationlight on light receiving surfaces of imaging elements of the imagingsection 109. Additionally, a zoom lens and a focus lens are configuredsuch that the positions of the lenses on an optical axis can be moved toadjust the magnification and focus of the captured image.

The imaging section 109 includes imaging elements and disposedsucceeding the lens unit 107. The observation light having passedthrough the lens unit 107 is concentrated on the light receivingsurfaces of the imaging elements and subjected to photoelectricconversion to generate an image signal corresponding to an observationimage. The image signal generated by the imaging section 109 is providedto the communication section 113.

As the imaging elements constituting the imaging section 109, forexample, CMOS (Complementary Metal Oxide Semiconductor)-type imagesensors are used that have a Bayer arrangement and that can capturecolor images. Note that imaging elements may be used that can support,for example, capturing of images of a high resolution of 4K or more. Animage of the affected site is obtained at high resolution to allow theoperator 167 to understand the state of the affected site in furtherdetail, enabling the surgery to proceed more smoothly.

Additionally, the imaging elements constituting the imaging section 109include a pair of imaging elements intended to acquire image signals forthe right eye and the left eye supporting 3D display. The 3D displayenables the operator 167 to accurately understand the depth of thebiological tissue in the affected site. Note that, in a case where theimaging section 109 includes multiple plates, a plurality of the lensunits 107 are provided in correspondence with the respective imagingelements.

Additionally, the imaging section 109 need not necessarily be providedin the camera head 105. For example, the imaging section 109 may beprovided in the lens barrel 103 located succeeding the objective lens.

The driving section 111 includes an actuator and moves the zoom lens andthe focus lens of the lens unit 107 a predetermined distance along theoptical axis under the control of the camera head control section 115.Accordingly, the magnification and focus of the captured image providedby the imaging section 109 may be appropriately adjusted.

The communication section 113 includes a communication apparatusconfigured to transmit and receive various types of information to andfrom the CCU 139. The communication section 113 transmits the imagesignal obtained from the imaging section 109, to the CCU 139 via thetransmission cable 165 as RAW data. At this time, to display thecaptured image of the affected site with low latency, the correspondingimage signal is preferably transmitted by optical communication. Thereason for the use of optical communication is as follows: during thesurgery, the operator 167 performs the surgery while observing the stateof the diseased site in the captured image, and thus, for more safe andreliable surgery, a moving image of the affected site is required to bedisplayed in real time whenever possible. In a case where opticalcommunication is performed, the communication section 113 is providedwith a photoelectric converting module that converts an electric signalinto an optical signal. After the image signal is converted into theoptical signal by the photoelectric converting module, the opticalsignal is transmitted to the CCU 139 via the transmission cable 165.

Additionally, the communication section 113 receives, from the CCU 139,a control signal for controlling driving of the camera head 105. Thecontrol signal includes information related to the imaging conditions,for example, information specifying a frame rate for the captured image,information specifying the value of exposure at the time of imaging,and/or information specifying the magnification and focus of thecaptured image. The communication section 113 provides the receivedcontrol signal to the camera head control section 115. Note that thecontrol signal from the CCU 139 may also be transmitted by opticalcommunication. In this case, the communication section 113 is providedwith a photoelectric converting module that converts an optical signalinto an electric signal. After the control signal is converted into theelectric signal, the electric signal is provided to the camera headcontrol section 115.

Note that the above-described imaging conditions such as the frame rate,the exposure value, the magnification, and the focus are automaticallyset by the control section 163 of the CCU 139 on the basis of theacquired image signal. In other words, what is called an AE (AutoExposure) function, an AF (Auto Focus) function, and an AWB (Auto WhiteBalance) function are provided in the endoscope 101.

The camera head control section 115 controls driving of the camera head105 on the basis of the control signal received from the CCU 139 via thecommunication section 113. For example, the camera head control section115 controls driving of the imaging elements of the imaging section 109on the basis of the information specifying the frame rate for thecaptured image and/or the information specifying the exposure at thetime of imaging. Additionally, for example, the camera head controlsection 115 appropriately moves the zoom lens and the focus lens of thelens unit 107 via the driving section 111 on the basis of theinformation specifying the magnification and focus of the capturedimage. The camera head control section 115 may further include afunction to store information used to identify the lens barrel 103 andthe camera head 105.

Note that, by disposing components such as the lens unit 107 and theimaging section 109 in a closed structure with high airtightness andhigh waterproofness, the camera head 105 can be made resistant toautoclave sterilization processing.

Now, the functional configuration of the CCU 139 will be described. Thecommunication section 159 includes a communication apparatus configuredto transmit and receive various types of information to and from thecamera head 105. The communication section 159 receives, from the camerahead 105, an image signal transmitted via the transmission cable 165. Atthis time, as described above, the image signal is suitably transmittedby optical communication. In this case, corresponding to the opticalcommunication, the communication section 159 is provided with aphotoelectric converting module that converts an optical signal into anelectric signal. The communication section 159 provides the imageprocessing section 161 with the image signal converted into the electricsignal.

Additionally, the communication section 159 transmits, to the camerahead 105, a control signal for controlling driving of the camera head105. The control signal may also be transmitted by opticalcommunication.

The image processing section 161 executes various types of imageprocessing on the image signal, which is RAW data transmitted from thecamera head 105. The image processing includes various types ofwell-known signal processing, for example, development processing, imagequality improvement processing (for example, band emphasis processing,super-resolution processing, NR (Noise Reduction) processing, and/orimage stabilization processing), and/or enlargement processing(electronic zoom processing). Additionally, the image processing section161 executes a detection processing on the image signal in order toperform AE, AF, and AWB.

The image processing section 161 includes a processor such as a CPU or aGPU, and the processor may operate in accordance with a predeterminedprogram to execute the above-described image processing or detectionprocessing. Note that, in a case where the image processing section 161includes a plurality of GPUs, the image processing section 161appropriately divides the information related to the image signal, andthe plurality of GPUs execute image processing in parallel.

The control section 163 performs various types of control related tocapturing of an image of the affected site by the endoscope 101 anddisplay of the captured image. For example, the control section 163generates a control signal for controlling driving of the camera head105. At this time, in a case where the imaging conditions have beeninput by the user, the control section 163 generates a control signal onthe basis of the input provided by the user. Alternatively, in a casewhere the endoscope 101 includes the AE function, AF function, and theAWB function, the control section 163 appropriately calculates theoptimal exposure value, focal length, and white balance in accordancewith the result of the detection processing executed by the imageprocessing section 161 and generates a control signal.

Additionally, the control section 163 cause the display apparatus 141 todisplays the image of the affected site on the basis of the image signalsubjected to the image processing by the image processing section 161.At this time, the control section 163 uses various recognizingtechniques to recognize various objects in the affected site image. Forexample, by detecting, for example, the shape or color of an edge of anobject included in the affected site image, the control section 163 canrecognize a surgical instrument such as forceps, a particular biologicalregion, bleeding, mist caused by the use of the energy treatmentinstrument 121, or the like. When causing the display apparatus 141 todisplay the image of the affected site, the control section 163 uses theresult of the recognition to cause the display apparatus 141 to displayvarious types of surgery assistance information on the image of theaffected site in a superimposed manner. The surgery assistanceinformation displayed in a superimposed manner is presented to theoperator 167 to enable the surgery to proceed safely and reliably.

The transmission cable 165 connecting the camera head 105 and the CCU139 together is an electric signal cable supporting communication ofelectric signals, an optical fiber supporting optical communication, ora composite cable corresponding to a combination of the electric signalcable and the optical fiber.

Here, in the illustrated example, wired communication is performed usingthe transmission cable 165. However, the communication between thecamera head 105 and the CCU 139 may be wireless. Wireless communicationbetween the camera head 105 and the CCU 139 eliminates a need to lay thetransmission cable 165 in the operating room. This may prevent movementof medical staff in the operating room from being hindered by thetransmission cable 165.

The example of the endoscopic surgery system 100 to which the techniqueaccording to the present disclosure may be applied has been described.Note that, here, the endoscopic surgery system 100 has been described byway of example but that the system to which the technique according tothe present disclosure may be applied is not limited to such an example.For example, the technique according to the present disclosure may beapplied to a soft endoscope system for examination or a microsurgerysystem.

2. Technical Features

Now, technical features of a medical observation system according to theembodiment of the present disclosure (in other words, a medical imagingsystem) will be described.

2.1. Basic Configuration

First, description will be given of an example of a basic functionalconfiguration of a control apparatus (in other words, an imageprocessing apparatus) executing image processing on an image captured bythe imaging section, such as the CCU 139 described above with referenceto FIG. 1. A particular focus is placed on sections of the controlapparatus that correct image blurring caused by camera shake or thelike. FIG. 3 is a block diagram illustrating an example of thefunctional configuration of the control apparatus in the medicalobservation system according to the present embodiment.

As illustrated in FIG. 3, a control apparatus 200 according to thepresent embodiment includes an image processing unit 210 and adetermining section 250.

The image processing unit 210 executes various types of analysisprocessing on an image signal (hereinafter also referred to as the“input image signal”) input from the imaging section at a predeterminedframe rate to detect image blurring caused by camera shake or the like.On the basis of the result of the detection, the image processing unit210 executes various types of processing on the input image signal tocorrect the image blurring.

Additionally, in the control apparatus 200 according to the presentembodiment, the determining section 250 controls application ornon-application of correction of image blurring and the degree of thecorrection (in other words, the intensity of the correction) in a casewhere the correction is to be applied, in accordance with predeterminedinput information such as detection results for various states orcircumstances. By way of a specific example, the determining section 250may control the application or non-application of the correction and thedegree of the correction by determining a coefficient for controllingthe degree of correction of image blurring in accordance with thepredetermined input information and notifying the image processing unit210 of the coefficient. Note that an example of control of the degree ofcorrection of image blurring in accordance with input information willbe separately described below in Examples in detail along with specificexamples of the input information. Additionally, in the descriptionbelow, the correction of image blurring may also be simply referred toas “blurring correction.” In addition, the determining section 250corresponds to an example of the “control section” controlling theapplication or non-application of correction of image blurring and thedegree of the correction.

Here, the configuration of the image processing unit 210 will bedescribed in further detail. As illustrated in FIG. 3, the imageprocessing unit 210 includes, for example, a first control section 211,a feature point extracting section 213, an estimating section 215, asecond control section 217, and a correction processing section 219.

The first control section 211 sets whether or not to apply the blurringcorrection to an input image signal on the basis of the control of thedetermining section 250. For example, according to the result of thedetermination made by the determining section 250 for whether or not toapply the blurring correction, the first control section 211 mayassociate the input image signal with a coefficient corresponding to theresult of the determination (that is, the coefficient depending onwhether or not to apply the blurring correction). By way of a morespecific example, the first control section 211 may set “1” as thecoefficient indicating whether or not to apply the blurring correctionin a case where the blurring correction is applied and set “0” as thecoefficient in a case where the blurring correction is not applied (thatis, the blurring correction is suppressed). Then, the first controlsection 211 associates the input image signal with the informationindicating whether or not to apply the blurring correction (for example,the coefficient indicating whether or not to apply the blurringcorrection) and outputs the resultant input image signal to the featurepoint extracting section 213.

Note that, in a case where selective switching between the applicationand non-application of the blurring correction is not necessary (thatis, in a case where the blurring correction is always applied), thefirst control section 211 need not be provided or may be disabled.

The feature point extracting section 213 applies image analysis to theinput image signal to extract characteristic portions from the image asfeature points on the basis of, for example, distribution of edges (forexample, wrinkles and patterns) and colors. The feature point extractingsection 213 then notifies the estimating section 215 of the input imagesignal and information related to the feature points extracted from theinput image signal.

The estimating section 215 estimates blurring of the entire image (morespecifically, the direction and amount of blurring) on the basis of theinput image signal and the information related to the feature points,the signal and information being output from the feature pointextracting section 213.

By way of a specific example, the estimating section 215 separates ascreen of the input image signal into blocks each with a predeterminedsize and compares each frame of the input image signal with a frame ofthe input image signal a predetermined number of frames before (forexample, the preceding frame) on a block-by-block basis to calculatemotion vectors in block units (hereinafter referred to as “local motionvectors”) and reliability of the motion vectors.

The estimating section 215 also integrates those of the local motionvectors in the respective blocks in each frame which have highreliability to determine a motion vector for the entire image in theframe (hereinafter also referred to as the “global motion vector”). Theestimating section 215 may also level global motion vectors for a numberof frames before the frame of interest to remove instantaneous errors.The estimating section 215 may also perform levelling using a number ofglobal motion vectors for a number of frames after the frame ofinterest.

The estimating section 215 also calculates a moving distance of theobjective lens in the imaging section on the basis of a detection resultfor acceleration or angular acceleration detected by various sensors(not illustrated) provided in the imaging section and the global motionvector and the reliability of the global motion vector. The estimatingsection 215 then estimates blurring of the entire image (for example,the direction and amount of image blurring) on the basis of thecalculated moving distance of the objective lens.

The estimating section 215 then associates the input image signal withinformation indicating the estimation result for the blurring of theentire image and outputs the resultant input image signal to the secondcontrol section 217.

The second control section 217 acquires the input image signal from theestimating section 215. The second control section 217 also controls thedegree of blurring correction (in other words, the correction intensity)applied to the input image signal on the basis of the control of thedetermining section 250 in accordance with the information associatedwith the acquired input image signal and indicating whether or not toapply the blurring correction.

By way of a specific example, the second control section 217 determinesthe correction intensity of the blurring correction (in other words, thecorrection amount of the blurring correction) on the basis of thecoefficient, associated with the input image signal, indicating whetheror not to apply the blurring correction (for example, “1” or “0”) andthe coefficient notified by the determining section 250 (that is, thecoefficient corresponding to the degree of the blurring correction). Byway of a specific example, the second control section 217 may calculatea correction coefficient (gain) indicative of the correction intensityof the blurring correction by multiplying the coefficient associatedwith the input image signal by the coefficient notified by thedetermining section 250. Accordingly, for example, in a case where theblurring correction is not applied to the input image signal, thecoefficient associated with the input image signal is “0,” and thus, theresult of calculation of the correction coefficient indicating thecorrection intensity is also “0.” As a result, the blurring correctionis not applied. Additionally, in a case where the blurring correction isapplied to the input image signal, the coefficient associated with theinput image signal is “1,” and thus, the correction coefficientindicative of the correction intensity of the blurring correction isdetermined according to the coefficient notified by the determiningsection 250.

The second control section 217 then outputs, to the correctionprocessing section 219, the input image signal and informationindicative of the correction intensity of the blurring correctiondetermined for the input image signal. Note that the informationindicative of the correction intensity of the blurring correctioncorresponds to information indicating whether or not to apply theblurring correction and the degree of the application of the blurringcorrection, and, for example, to information indictive of theabove-described correction coefficient for the blurring correction byway of example.

The correction processing section 219 acquires the input image signalfrom the second control section 217, and applies the blurring correctionto the input image signal in accordance with various types ofinformation associated with the input image signal.

Specifically, the correction processing section 219 outputs, to acomponent located succeeding the correction processing section 219 (forexample, the display apparatus), as an output image signal, an imagesignal obtained by cutting out, from the entire region (effective pixelarea) of the input image signal, a cutout area smaller in size than theeffective pixel area. At this time, the correction processing section219 shifts the position of the cutout area by a shift amountcorresponding to the blurring of the entire image to correct theblurring. Note that the position of the cutout area (in other words, theshift amount corresponding to the blurring of the entire image) may bedetermined on the basis of information indicating the estimation resultfor the blurring of the entire image associated with the input imagesignal (that is, the information indicative of the direction and amountof the blurring).

Additionally, the correction processing section 219 may control thedegree of the blurring correction applied to the input image signal onthe basis of the information associated with the input image signal andindicating the correction intensity of the blurring correction. By wayof a specific example, the correction processing section 219 maycontrol, in accordance with the correction intensity, a threshold fordetermining whether or not to apply the blurring correction. In thiscase, the correction processing section 219 may apply the blurringcorrection in a case where the amount of blurring is smaller than thethreshold, and provides control such that the threshold increasesconsistently with the correction intensity (that is, the thresholdincreases so as to include more significant blurring within the range ofblurring to be corrected).

The correction processing section 219 may also control the correctionamount of the blurring correction on the basis of the informationindicative of the correction intensity of the blurring correction. Byway of a specific example, the correction processing section 219 maylimit the correction amount of the blurring correction (that is, theshift amount for the position of the cutout area) such that thecorrection amount decreases consistently with correction intensity.

The example of the basic functional configuration of the controlapparatus executing image processing on the image captured by theimaging section has been described above with reference to FIG. 3, witha particular focus on the sections of the control apparatus that correctimage blurring caused by camera shake or the like.

2.2. Processing

Now, an example of a flow of a sequence of processing by the controlapparatus executing image processing on the image captured by theimaging section will be described with reference to FIG. 4, with aparticular focus on the sections of the control apparatus that correctimage blurring caused by camera shake or the like. FIG. 4 is a flowchartillustrating an example of a flow of a sequence of processing by thecontrol apparatus in the medical observation system according to thepresent embodiment.

First, the control apparatus 200 acquires an image signal output fromthe imaging section at a predetermined frame rate (S101). At this time,the control apparatus 200 (first control section 211) may associate,with the acquired image signal (that is, the input image signal),information indicative of the application or non-application of theblurring correction determined in accordance with the predeterminedinput information.

The control apparatus 200 (feature point extracting section 213) thenapplies image analysis to the input image signal to extractcharacteristic portions from the image as feature points (S103).

The control apparatus 200 (estimating section 215) also estimates theblurring of the entire image on the basis of the result of extraction offeature points. By way of a specific example, the control apparatus 200separates the screen of the input image signal into blocks each with apredetermined size and compares each frame of the input image signalwith a frame of the input image signal a predetermined number of framesbefore on a block-by-block basis to calculate local motion vectors andreliability of the local motion vectors. The control apparatus 200 alsointegrates those of the local motion vectors in the respective blocks ineach frame which have high reliability to determine the global motionvector. The control apparatus 200 also calculates the moving distance ofthe objective lens in the imaging section on the basis of the detectionresult for the acceleration or angular acceleration detected by thevarious sensors provided in the imaging section and the global motionvector and the reliability of the global motion vector. The controlapparatus 200 then estimates the blurring of the entire image (forexample, the direction and amount of image blurring) on the basis of thecalculated moving distance of the objective lens (S105).

The control apparatus 200 (second control section 217) then controls thecorrection intensity of the blurring correction (that is, the degree ofthe blurring correction) applied to the input image signal in accordancewith the predetermined input information. By way of a specific example,the control apparatus 200 may determine, in accordance with the inputinformation, the coefficient indicative of the correction intensity ofthe blurring correction applied to the input image signal (S200).

The control apparatus 200 (correction processing section 219) thenapplies the blurring correction to the input image signal on the basisof the estimation result for the blurring of the entire image and thecorrection intensity corresponding to the predetermined inputinformation, and outputs the corrected image signal to the componentlocated succeeding the correction processing section 219 (for example,the display apparatus), as the output image signal. Note that the methodfor the blurring correction is as described above and will thus not bedescribed below in detail (S107).

The example of the flow of the sequence of processing by the controlapparatus executing image processing on the image captured by theimaging section has been described with reference to FIG. 4, with aparticular focus on the sections of the control apparatus that correctimage blurring caused by camera shake or the like.

2.3. Examples

Now, by way of examples of the medical observation system according tothe present embodiment, an example of control related to the correctionof image blurring performed by the control apparatus 200 as describedabove with reference to FIG. 3 will be described below, along withspecific examples of the input information.

Example 1: Control According to Zoom Magnification

First, with reference to FIG. 5 and FIG. 6, Example 1 will be describedthat is an example of control using, as input information, informationindicative of a zoom magnification of the imaging section and related tocorrection of image blurring according to the zoom magnification. FIG. 5and FIG. 6 are descriptive diagrams illustrating an example of thecontrol related to the correction of image blurring performed by acontrol apparatus according to Example 1. Note that in this description,the control apparatus according to the present example may be referredto as a “control apparatus 200 a” so as to be distinguished from controlapparatuses according to the other examples.

For example, FIG. 5 illustrates an example of a functional configurationof the control apparatus 200 a according to Example 1. Note that thecontrol apparatus 200 a differs from the control apparatus 200 describedwith reference to FIG. 3 mainly in the operation of each of adetermining section 250 a, and a first control section 211 a and asecond control section 217 a of an image processing unit 210 a. Thus,for operations of the control apparatus 200 a, FIG. 3 will be describedwith focus on sections different from the corresponding sections of thecontrol apparatus 200. Sections of the control apparatus 200 a that aresubstantially similar to the corresponding sections of the controlapparatus 200 will not be described below in detail.

For example, the determining section 250 a acquires information relatedto setting of the zoom magnification, from the imaging section as inputinformation, and determines whether or not to apply blurring correctionin accordance with the input information. In a case of applying theblurring correction, the determining section 250 a determines acoefficient for controlling the degree of the application (that is, thecorrection intensity). That is, the first control section 211 a controlswhether or not to apply the blurring correction on the basis of theresult of the determination made by the determining section 250 aaccording to the zoom magnification. Additionally, the second controlsection 217 a determines a correction coefficient indicative of thecorrection intensity of the blurring correction on the basis of theresult of the determination made by the determining section 250 aaccording to the zoom magnification.

Now, the example of the control related to the correction of imageblurring according to the zoom magnification will be described infurther detail with reference to FIG. 6. FIG. 6 is a graph illustratingan example of a relationship between the zoom magnification and thecoefficient for controlling the correction intensity of the blurringcorrection in the present example. In FIG. 6, the abscissa axisindicates the zoom magnification, and the ordinate axis indicates thecoefficient for controlling the correction intensity. Note that, in thepresent example, the coefficient for controlling the correctionintensity is determined to range from not less than 0 to not more than1, and the coefficient of 0 is substantially similar to thenon-application of the blurring correction.

As illustrated in FIG. 6, in the present example, the coefficient forcontrolling the correction intensity is set to decrease and increaseconsistently with zoom magnification. Additionally, a zoom magnificationsmaller than or equal to a predetermined threshold may disable(suppress) the blurring correction.

By way of a specific example, a zoom magnification set to a lower valueincreases the range of an image captured (that is, the visual fieldrange). When the correction intensity of the blurring correction isincreased under these circumstances, for example, in a case where amotion vector indicative of motion of a surgical instrument such asintensely shaking forceps is employed as a global motion vector, abird's-eye view image corresponding to a background may shake intenselyto make an observation target such as a living organism difficult toobserve. Thus, in such a case, the observation target such as the livingorganism may be made easier to observe by, for example, reducing thecorrection intensity of the blurring correction or disabling(suppressing) the blurring correction.

Additionally, by way of another example, in a case where the correctionintensity of the blurring correction is increased with the zoommagnification set lower, when a cutout area is cut out from theeffective pixel area, the regions removed as portions other than thecutout area (for example, end sides of the effective pixel area) tend tobe larger. Thus, the blurring correction is applied even in a case wherethe range imaged by the imaging section is intentionally moved as in acase where a scope of the endoscope apparatus or the like isintentionally moved. As a result, the observation target such as theliving organism may become difficult to observe. Even in such a case,the observation target such as the living organism may be made easier toobserve by, for example, reducing the correction intensity of theblurring correction or disabling (suppressing) the blurring correction.

Note that the relationship between the zoom magnification and thecoefficient for controlling the correction intensity of the blurringcorrection, illustrated in FIG. 6, is only an example. That is, therelationship between the zoom magnification and the coefficient is notnecessarily limited to the example illustrated in FIG. 6 as long ascontrol is provided such that the correction intensity of the blurringcorrection is increased consistently with zoom magnification.

With reference to FIG. 5 and FIG. 6, Example 1 has been described, whichis an example of the control using, as input information, theinformation indicative of the zoom magnification of the imaging sectionand related to the correction of image blurring according to the zoommagnification.

Example 2: Control According to Working Distance to Subject

Now, with reference to FIG. 7 and FIG. 8, Example 2 will be describedthat is an example of control using, as input information, informationindicative of a working distance to a subject and related to correctionof image blurring according to the working distance. FIG. 7 and FIG. 8are descriptive diagrams illustrating an example of control related tocorrection of image blurring performed by a control apparatus accordingto Example 2. Note that, in this description, the control apparatusaccording to the present example may be referred to as a “controlapparatus 200 b” so as to be distinguished from control apparatusesaccording to the other examples. Additionally, the working distance isintended to indicate a distance between the objective lens of theimaging section and a subject (for example, a living organism to beobserved).

For example, FIG. 7 illustrates an example of a functional configurationof the control apparatus 200 b according to Example 2. Note that thecontrol apparatus 200 b differs from the control apparatus 200 describedwith reference to FIG. 3 mainly in the operation of each of adetermining section 250 b, and a first control section 211 b and asecond control section 217 b of an image processing unit 210 b. Thus,for operations of the control apparatus 200 b, FIG. 3 will be describedwith focus on sections different from the corresponding sections of thecontrol apparatus 200. Sections of the control apparatus 200 b that aresubstantially similar to the corresponding sections of the controlapparatus 200 will not be described below in detail.

For example, the determining section 250 b acquires, as inputinformation, information indicative of a detection result for thedistance between the objective lens of the imaging section and thesubject (that is, the working distance), the detection result beingprovided by a distance measuring sensor or the like. Additionally, byway of another example, the determining section 250 b may acquire, asinput information, information indicative of the distance between theobjective lens of the imaging section and the subject in an imagecaptured by the imaging section, the distance being calculated byanalyzing the image. The determining section 250 b determines whether ornot to apply the blurring correction in accordance with the acquiredinput information (that is, information indicative of the workingdistance), and in a case where the blurring correction is applied,determines a coefficient for controlling the degree of the application(that is, the correction intensity). That is, the first control section211 b controls whether or not to apply the blurring correction on thebasis of the result of determination made by the determining section 250b according to the working distance. Additionally, the second controlsection 217 b determines a correction coefficient indicative of thecorrection intensity of the blurring correction on the basis of theresult of the determination made by the determining section 250 baccording to the working distance.

Here, with reference to FIG. 8, the example of the control related tothe correction of image blurring according to the working distance willbe described in further detail. FIG. 6 is a graph illustrating anexample of a relationship between the working distance and thecoefficient for controlling the correction intensity of the blurringcorrection. In FIG. 6, the abscissa axis indicates the zoommagnification, and the ordinate axis indicates the coefficient forcontrolling the correction intensity. Note that, in the present example,the coefficient for controlling the correction intensity is determinedto range from not less than 0 to not more than 1, and the coefficient of0 is substantially similar to the non-application of the blurringcorrection.

As illustrated in FIG. 8, in the present example, the coefficient forcontrolling the correction intensity is set to decrease and increaseconsistently with working distance (that is, to decrease and increaseconsistently with the distance between the objective lens of the imagingsection and the subject). Additionally, a working distance larger thanor equal to a threshold may disable (suppress) the blurring correction.

By way of a specific example, a decrease in working distance (that is,in distance between the objective lens of the imaging section and thesubject) reduces the range of the image captured (that is, the visualfield range), increasing the amount of blurring of the entire imagerelative to the moving distance of the objective lens of the imagingsection. Thus, the observation target such as the living organism may bemade easier to observe by, for example, increasing the correctionintensity of the blurring correction (that is, increasing thecoefficient for controlling the correction intensity) with decreasingworking distance.

Note that the relationship between the working distance and thecoefficient for controlling the correction intensity of the blurringcorrection, illustrated in FIG. 8, is only an example. That is, therelationship between the working distance and the coefficient is notnecessarily limited to the example illustrated in FIG. 8 as long ascontrol is provided such that the correction intensity of the blurringcorrection is increased consistently with decreasing working distance.

With reference to FIG. 7 and FIG. 8, Example 2 has been described, whichis an example of the control using, as input information, theinformation indicative of the working distance to the subject andrelated to the correction of image blurring according to the workingdistance.

Example 3: Control According to Operative Duration

Now, with reference to FIG. 9 and FIG. 10, Example 3 will be describedthat is an example of control using, as input information, informationindicative of an operative duration and related to correction of imageblurring according to the operative duration. FIG. 9 and FIG. 10 aredescriptive diagrams illustrating an example of control related tocorrection of image blurring performed by a control apparatus accordingto Example 3. Note that, in this description, the control apparatusaccording to the present example may be referred to as a “controlapparatus 200 c” so as to be distinguished from control apparatusesaccording to the other examples.

For example, FIG. 9 illustrates an example of a functional configurationof the control apparatus 200 c according to Example 3. Note that aprecondition for the present example is that image blurring is to becorrected. Thus, in the control apparatus 200 c, an image processingunit 210 c includes no component corresponding to the first controlsection 211 described above with reference to FIG. 3. Additionally, thecontrol apparatus 200 c differs from the control apparatus 200 describedwith reference to FIG. 3 mainly in the operation of each of adetermining section 250 c, and a second control section 217 b of theimage processing unit 210 c. Thus, for operations of the controlapparatus 200 c, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 c that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

For example, the determining section 250 c acquires, as inputinformation, information indicative of a measurement result for theoperative duration obtained from a clocking section or the like, anddetermines a coefficient for controlling the degree of application ofthe blurring correction (that is, the correction intensity) inaccordance with the acquired input information (that is, the informationindicative of the operative duration). That is, the second controlsection 217 c determines a correction coefficient indicative of thecorrection intensity of the blurring correction on the basis of theresult of the determination made by the determining section 250 caccording to the operative duration.

Here, the example of the control related to the correction of imageblurring according to the operative duration will be described infurther detail with reference to FIG. 10. FIG. 10 is a graphillustrating an example of a relationship between the operative durationand the coefficient for controlling the correction intensity of theblurring correction in the present example. In FIG. 10, the abscissaaxis indicates the operative duration, and the ordinate axis indicatesthe coefficient for controlling the correction intensity. Note that, inthe present example, the coefficient for controlling the correctionintensity is determined to range from not less than 0 to not more than1, and the coefficient of 0 is substantially similar to thenon-application of the blurring correction.

As illustrated in FIG. 10, in the present example, the coefficient forcontrolling the correction intensity is set to increase consistentlywith the operative duration in a case where the operative durationexceeds a predetermined threshold.

By way of a specific example, in a case where the endoscope is assumedto be held by a scopist, it is assumed that an extended operativeduration increases the likelihood that the endoscope is shaken becausethe extended operative duration makes the scopist more and more tired.In light of these circumstances, control may be provided such that thecorrection intensity of the blurring correction is increasedconsistently with operative duration.

Note that the relationship between the operative duration and thecoefficient for controlling the correction intensity of the blurringcorrection, illustrated in FIG. 10, is only an example. That is, therelationship between the operative duration and the coefficient is notnecessarily limited to the example illustrated in FIG. 10 as long ascontrol is provided such that the correction intensity of the blurringcorrection is increased consistently with operative duration.

With reference to FIG. 9 and FIG. 10, Example 3 has been described,which is an example of the control using, as input information, theinformation indicative of the operative duration and related to thecorrection of image blurring according to the operative duration.

Example 4: Control According to Vibration of Operating Table

Now, with reference to FIG. 11 and FIG. 12, Example 4 will be describedthat is an example of control using, as input information, informationindicating a detection result for vibration of an operating table andrelated to correction of image blurring according to the vibration ofthe operating table. FIG. 11 and FIG. 12 are descriptive diagramsillustrating an example of control related to correction of imageblurring performed by a control apparatus according to Example 4. Notethat, in this description, the control apparatus according to thepresent example may be referred to as a “control apparatus 200 d” so asto be distinguished from control apparatuses according to the otherexamples.

For example, FIG. 11 illustrates an example of a functionalconfiguration of the control apparatus 200 d according to Example 4.Additionally, the control apparatus 200 d differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 d, and a first control section 211d and a second control section 217 d of an image processing unit 210 d.Thus, for operations of the control apparatus 200 d, FIG. 3 will bedescribed with focus on sections different from the correspondingsections of the control apparatus 200. Sections of the control apparatus200 d that are substantially similar to the corresponding sections ofthe control apparatus 200 will not be described below in detail.

For example, the determining section 250 d acquires, as inputinformation, information indicative of the detection result for thevibration of the operating table detected by various sensors such as avibration sensor. The determining section 250 d then determines whetheror not to apply the blurring correction in accordance with the acquiredinput information (that is, information indicative of the vibration ofthe operating table), and in a case where the blurring correction isapplied, determines a coefficient for controlling the degree ofapplication of the blurring correction (that is, the correctionintensity). That is, the first control section 211 d controls whether ornot to apply the blurring correction on the basis of the result of thedetermination made by the determining section 250 d according to thevibration of the operating table. Additionally, the second controlsection 217 d determines the correction coefficient indicative of thecorrection intensity of the blurring correction on the basis of theresult of the determination made by the determining section 250 daccording to the vibration of the operating table.

Here, with reference to FIG. 12, detailed description will be given ofan example of control related to correction of image blurring accordingto the vibration of the operating table. FIG. 6 is a graph illustratingthe example of the relationship between the vibration of the operatingtable and the coefficient for controlling the correction intensity ofthe blurring correction in the present example. In FIG. 6, the abscissaaxis indicates the magnitude of vibration of the operating table, andthe ordinate axis indicates the coefficient for controlling thecorrection intensity. Note that, in the present example, the coefficientfor controlling the correction intensity is determined to range from notless than 0 to not more than 1, and the coefficient of 0 issubstantially similar to the non-application of the blurring correction.

As illustrated in FIG. 12, in the present example, the coefficient forcontrolling the correction intensity is set to decrease with increasingmagnitude of vibration of the operating table in a case where themagnitude of the vibration exceeds a predetermined threshold.Additionally, in a case where the magnitude of vibration of theoperating table is larger than or equal to the threshold, the blurringcorrection may be disabled (suppressed).

That is, in the present example, the correction intensity of theblurring correction is reduced with increasing magnitude of vibration ofthe operating table to inhibit exertion of adverse effect of theblurring correction.

Note that the relationship between the magnitude of vibration of theoperating table and the coefficient for controlling the correctionintensity of the blurring correction, illustrated in FIG. 12, is only anexample. That is, the relationship between the magnitude of vibration ofthe operating table and the coefficient for controlling the correctionintensity of the blurring correction is not necessarily limited to theexample illustrated in FIG. 12 as long as control is provided such thatthe correction intensity of the blurring correction is reduced withincreasing magnitude of vibration of the operating table.

With reference to FIG. 11 and FIG. 12, Example 4 has been described,which is an example of the control using, as input information, theinformation indicating the detection result for the vibration of theoperating table and related to the correction of image blurringaccording to the operative duration.

Example 5: Control According to Occupancy of Surgical Instrument inScreen

Now, with reference to FIGS. 13 to 15, Example 5 will be described thatis an example of control using, as input information, informationindicative of occupancy of a surgical instrument in a screen and relatedto correction of image blurring according to the occupancy of thesurgical instrument. FIGS. 13 to 15 are descriptive diagramsillustrating an example of control related to correction of imageblurring performed by a control apparatus according to Example 5. Notethat, in this description, the control apparatus according to thepresent example may be referred to as a “control apparatus 200 e” so asto be distinguished from control apparatuses according to the otherexamples.

For example, FIG. 13 illustrates an example of a functionalconfiguration of the control apparatus 200 e according to Example 5.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 e, an imageprocessing unit 210 e includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 e differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 e, and a second control section 217e of the image processing unit 210 e. Thus, for operations of thecontrol apparatus 200 e, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 e that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

The determining section 250 e acquires, as input information,information indicative of the occupancy of a surgical instrument in ascreen recognized on the basis of an object recognizing technique or thelike.

For example, FIG. 14 is a descriptive diagram illustrating an example ofprocessing related to recognition of the occupancy of the surgicalinstrument in the screen. Specifically, in an upper diagram of FIG. 14,reference sign V301 denotes an example of an image of a living organism(image of a diseased site or the like) captured via the endoscope or thelike. That is, in the image V301 surgical instruments such as forceps311 and 313 and gauze 315 as well as a living organism to be observed inthe screen are captured.

Additionally, a lower diagram of FIG. 14 illustrates an example of arecognition result for objects other than the living organism (forexample, the surgical instruments) captured in the image V301, therecognition result being based on scene recognition processing executed,utilizing the object recognizing technique or the like, on the imageV301 illustrated in the upper diagram. Specifically, regions denoted byreference signs V311 and V313 schematically represent an example of arecognition result for regions occupied by the forceps 311 and 313 inthe screen. Additionally, a region denoted by reference sign V315schematically represents an example of a recognition result for a regionoccupied by the gauze 315 in the screen. Additionally, reference signV317 schematically represents an example of a recognition result for aregion occupied by the lens barrel of the endoscope or the like in thescreen V301.

That is, by utilizing the object recognizing technique or the like torecognize surgical instruments and the like captured in the image, forexample, the occupancy of objects such as surgical instruments in thescreen (in other words, the objects other than the living organism) canbe calculated as illustrated in the lower diagram in FIG. 14.

Then, the determining section 250 e determines a coefficient forcontrolling the degree of application of the blurring correction (thatis, the correction intensity) in accordance with the acquired inputinformation (that is, the information indicative of the occupancy of thesurgical instruments in the screen). That is, as illustrated in FIG. 13,the second control section 217 e determines a correction coefficientindicative of the correction intensity of the blurring correction on thebasis of the result of the determination made by the determining section250 e according to the occupancy of the surgical instruments in thescreen.

Here, with reference to FIG. 15, detailed description will be given ofan example of control related to correction of image blurring accordingto the occupancy of the surgical instruments in the screen. FIG. 15 is agraph illustrating the example of the relationship between the occupancyof the surgical instruments in the screen and the coefficient forcontrolling the correction intensity of the blurring correction in thepresent example. In FIG. 10, the abscissa axis indicates the occupancyof the surgical instruments in the screen, and the ordinate axisindicates the coefficient for controlling the correction intensity. Notethat, in the present example, the coefficient for controlling thecorrection intensity is determined to range from not less than 0 to notmore than 1, and the coefficient of 0 is substantially similar to thenon-application of the blurring correction.

As illustrated in FIG. 15, in the present example, the coefficient forcontrolling the correction intensity is set to decrease with increasingoccupancy of the surgical instruments in the screen in a case where theoccupancy exceeds a predetermined threshold.

By way of a specific example, an increased occupancy of the surgicalinstruments in the screen increases the ratio of a region where theliving organism to be observed is shielded by the surgical instruments(that is, the subjects other than the living organism). Under suchcircumstances, in a case where feature points are extracted only fromthe living organism region, the number of feature points extracteddecreases consistently with the ratio of the living organism region(that is, decreases with increasing ratio of the region shielded by thesurgical instruments and the like). In such a case, reliability ofmotion vectors calculated on the basis of the result of extraction offeature points decreases to also reduce reliability of a correctionvalue (for example, a shift amount) for the blurring correction based onthe motion vectors. Note that, due to such properties, for example, theuse of the result of object recognition is desirably avoided in settingthe range of extraction of feature points. Additionally, even in a casewhere the extraction of feature points covers portions other than theliving organism region, the coverage includes a large number of regionsother than the living organism, thus reducing the reliability of motionvectors calculated on the basis of the result of extraction of featurepoints. This may result in a decrease in the reliability of thecorrection value for the blurring correction based on the motionvectors.

Accordingly, it is assumed that, in a case where the correctionintensity of the blurring correction is increased with the reliabilityof the correction value for the blurring correction reduced, theblurring correction fails to be applied in an appropriate manner, makingthe observation target such as the living organism difficult to observe.Thus, the observation target such as the living organism can beprevented from becoming difficult to observe by, for example, reducingthe correction intensity of the blurring correction with increasingoccupancy of the regions other than the living organism (for example,the occupancy of the surgical instruments) in the screen.

Note that the relationship between the occupancy of the surgicalinstruments in the screen and the coefficient for controlling thecorrection intensity of the blurring correction, illustrated in FIG. 15,is only an example. That is, the relationship between the occupancy ofthe surgical instruments in the screen and the coefficient is notnecessarily limited to the example illustrated in FIG. 15 as long ascontrol is provided such that the correction intensity of the blurringcorrection is reduced with increasing occupancy. Additionally, theexample of the case of reduction in the correction intensity of theblurring correction has been described above. However, control may beprovided such that an increased occupancy of the surgical instrumentsdisables (suppresses) the blurring correction.

With reference to FIGS. 13 to 15, Example 5 has been described, which isan example of the control using, as input information, the informationindicative of the occupancy of the surgical instruments in the screenand related to the correction of image blurring according to theoccupancy of the surgical instruments.

Example 6: Control According to Observation Mode

Now, with reference to FIG. 16, Example 6 will be described that is anexample of control using, as input information, information indicativeof an observation mode and related to correction of image blurringaccording to the observation mode. FIG. 16 is a descriptive diagramillustrating an example of control related to correction of imageblurring performed by a control apparatus according to Example 6. Notethat, in this description, the control apparatus according to thepresent example may be referred to as a “control apparatus 200 f” so asto be distinguished from control apparatuses according to the otherexamples. Additionally, the observation mode refers to, for example, amode corresponding to observation method, for example, white lightobservation, special light observation such as infrared observation orfluorescent observation, or PDD (Photodynamic diagnosis).

For example, FIG. 16 illustrates an example of a functionalconfiguration of the control apparatus 200 f according to Example 6.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 f, an imageprocessing unit 210 f includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 f differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 f, and a second control section 217f of the image processing unit 210 f. Thus, for operations of thecontrol apparatus 200 f, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 f that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

The determining section 250 f, for example, acquires, as inputinformation, information related to setting of the observation mode, anddetermines a coefficient for controlling the degree of application ofthe blurring correction (that is, the correction intensity) inaccordance with the acquired input information (that is, the informationindicative of the observation mode). That is, the second control section217 f determines a correction coefficient indicative of the correctionintensity of the blurring correction on the basis of the result of thedetermination made by the determining section 250 f according to theobservation mode.

Specifically, the special light observation tends to involve loweraccuracy for calculation of motion vectors than the white lightobservation. Thus, under circumstances such as the special lightobservation where the frame rate is reduced, the correction intensity ofthe blurring correction may be set higher than in the white lightobservation so as to allow the observation target such as the livingorganism to be observed as is the case with the white light observation.

With reference to FIG. 16, Example 6 has been described, which is anexample of the control using, as input information, the informationindicative of the observation mode and related to the correction ofimage blurring according to the observation mode.

Example 7: Control According to Distance Between Monitor and Operator

Now, with reference to FIG. 17 and FIG. 18, Example 7 will be describedthat is an example of control using, as input information, informationindicative of a distance between a monitor and the operator and relatedto correction of image blurring according to the distance. FIG. 17 andFIG. 18 are descriptive diagrams illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 7. Note that, in this description, the controlapparatus according to the present example may be referred to as a“control apparatus 200 g” so as to be distinguished from controlapparatuses according to the other examples.

For example, FIG. 17 illustrates an example of a functionalconfiguration of the control apparatus 200 g according to Example 7.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 g, an imageprocessing unit 210 g includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 g differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 g, and a second control section 217g of the image processing unit 210 g. Thus, for operations of thecontrol apparatus 200 g, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 g that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

The determining section 250 g acquires, as input information,information related to a measurement result for a distance between theoperator and a predetermined monitor (for example, a main monitor), anddetermines a coefficient for controlling the degree of application ofthe blurring correction (that is, the correction intensity) inaccordance with the acquired input information (that is, the informationindicative of the distance between the operator and the monitor). Thatis, the second control section 217 g determines a correction coefficientindicative of the correction intensity of the blurring correction on thebasis of the result of the determination made by the determining section250 g according to the distance between the operator and the monitor.Note that a method for acquiring the information indicative of thedistance between the operator and the monitor is not particularlylimited as long as the method allows the information to be acquired. Byway of a specific example, the distance between the operator and themonitor may be measured (calculated) on the basis of an image capturedwith a camera for a surgical field or the like and illustrating theoperator and monitor. Additionally, by way of another example, theinformation indicative of the distance between the operator and themonitor may be acquired on the basis of the detection result from thedistance measuring sensor and the like.

Here, with reference to FIG. 18, detailed description will be given ofan example of control related to correction of image blurring accordingto the distance between the operator and the monitor. FIG. 18 is a graphillustrating the example of the relationship between the distancebetween the operator and the monitor and the coefficient for controllingthe correction intensity of the blurring correction in the presentexample. In FIG. 18, the abscissa axis indicates the distance betweenthe operator and the monitor, and the ordinate axis indicates thecoefficient for controlling the correction intensity. Note that, in thepresent example, the coefficient for controlling the correctionintensity is determined to range from not less than 0 to not more than1, and the coefficient of 0 is substantially similar to thenon-application of the blurring correction.

As illustrated in FIG. 18, in the present example, the coefficient forcontrolling the correction intensity is set to increase with decreasingdistance between the operator and the monitor (that is, increasingcloseness from the operator to the monitor). Additionally, in a casewhere the distance between the operator and the monitor exceeds apredetermined threshold, the coefficient for controlling the correctionintensity may be maintained at a predetermined value.

By way of a specific example, the degree of perception of image blurringobtained by the operator relatively increases with decreasing distancebetween the operator and the monitor. In light of such circumstances,control may be provided such that the correction intensity of theblurring correction is increased with decreasing distance between theoperator and the monitor.

Note that the relationship between the distance between the operator andthe monitor and the coefficient for controlling the correction intensityof the blurring correction, illustrated in FIG. 18, is only an example.That is, the relationship between the distance between the operator andthe monitor and the coefficient is not necessarily limited to theexample illustrated in FIG. 18 as long as control is provided such thatthe correction intensity of the blurring correction is increased withdecreasing distance between the operator and the monitor.

With reference to FIG. 17 and FIG. 18, Example 7 has been described,which is an example of the control using, as input information, theinformation indicative of the distance between the operator and themonitor and related to the correction of image blurring according to thedistance.

Example 8: Control According to Monitor Size

Now, with reference to FIG. 19 and FIG. 20, Example 8 will be describedthat is an example of control using, as input information, informationindicative of a monitor size of a predetermined monitor and related tocorrection of image blurring according to the monitor size. FIG. 19 andFIG. 20 are descriptive diagrams illustrating an example of controlrelated to correction of image blurring performed by a control apparatusaccording to Example 8. Note that, in this description, the controlapparatus according to the present example may be referred to as a“control apparatus 200 h” so as to be distinguished from controlapparatuses according to the other examples.

For example, FIG. 19 illustrates an example of a functionalconfiguration of the control apparatus 200 h according to Example 8.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 h, an imageprocessing unit 210 h includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 h differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 h, and a second control section 217h of the image processing unit 210 h. Thus, for operations of thecontrol apparatus 200 h, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 h that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

The determining section 250 h acquires, as input information,information indicative of the monitor size of the predetermined monitor(for example, the main monitor), and determines a coefficient forcontrolling the degree of application of the blurring correction (thatis, the correction intensity) in accordance with the acquired inputinformation (that is, the information indicative of the monitor size).That is, the second control section 217 h determines a correctioncoefficient indicative of the correction intensity of the blurringcorrection on the basis of the result of the determination made by thedetermining section 250 h according to the monitor size. Note that amethod for acquiring the information indicative of the monitor size ofthe predetermined monitor is not particularly limited as long as themethod allows the information to be acquired. By way of a specificexample, the information indicative of the monitor size of thepredetermined monitor may be acquired from the monitor itself on whichthe image is displayed.

Here, with reference to FIG. 20, detailed description will be given ofan example of control related to correction of image blurring accordingto the monitor size. FIG. 20 is a graph illustrating the example of therelationship between the monitor size and the coefficient forcontrolling the correction intensity of the blurring correction in thepresent example. In FIG. 20, the abscissa axis indicates the monitorsize, and the ordinate axis indicates the coefficient for controllingthe correction intensity. Note that, in the present example, thecoefficient for controlling the correction intensity is determined torange from not less than 0 to not more than 1, and the coefficient of 0is substantially similar to the non-application of the blurringcorrection.

As illustrated in FIG. 20, in the present example, the coefficient forcontrolling the correction intensity is set to increase consistentlywith monitor size. Additionally, in a case where the monitor sizeexceeds a predetermined threshold, the coefficient for controlling thecorrection intensity may be maintained at a predetermined value.

By way of a specific example, an increased monitor size increases thesize of the image displayed on the monitor, relatively increasing a blurwidth in the image. In light of such circumstances, control may beprovided such that the correction intensity of the blurring correctionis increased consistently with monitor size.

Note that the relationship between the monitor size and the coefficientfor controlling the correction intensity of the blurring correction,illustrated in FIG. 20, is only an example. That is, the relationshipbetween the monitor size and the coefficient is not necessarily limitedto the example illustrated in FIG. 20 as long as control is providedsuch that the correction intensity of the blurring correction isincreased consistently with monitor size.

With reference to FIG. 19 and FIG. 20, Example 8 has been described,which is an example of the control using, as input information, theinformation indicative of the monitor size of the predetermined monitorand related to the correction of image blurring according to the monitorsize.

Example 9: Control According to State of Surgical Instrument

Now, with reference to FIGS. 21 to 23, Example 9 will be described thatis an example of control using, as input information, informationindicative of a state of a surgical instrument and related to correctionof image blurring according to the state of the surgical instrument.FIGS. 21 to 23 are descriptive diagrams illustrating an example ofcontrol related to correction of image blurring performed by a controlapparatus according to Example 9. Note that, in this description, thecontrol apparatus according to the present example may be referred to asa “control apparatus 200 i” so as to be distinguished from controlapparatuses according to the other examples.

For example, FIG. 21 illustrates an example of a functionalconfiguration of the control apparatus 200 i according to Example 9.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 i, an imageprocessing unit 210 i includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 i differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 i, and a second control section 217i of the image processing unit 210 i. Thus, for operations of thecontrol apparatus 200 i, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 i that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

The determining section 250 i acquires, as input information,information indicative of the state of a surgical instrument, forexample, a stapler, an energy device, a suture needle and thread, or adebrider, and determines a coefficient for controlling the degree ofapplication of the blurring correction (that is, the correctionintensity) in accordance with the acquired input information (that is,the information indicative of the state of the surgical instrument).That is, the second control section 217 i determines a correctioncoefficient indicative of the correction intensity of the blurringcorrection on the basis of the result of the determination made by thedetermining section 250 i according to the state of the surgicalinstrument. Note that a method for acquiring the information indicativeof the state of the surgical instrument is not particularly limited aslong as the method allows the information to be acquired. By way of aspecific example, the information indicative of the state of thesurgical instrument may be acquired from the target surgical instrumentor the control section controlling the surgical instrument.Additionally, by way of another example, the state of the surgicalinstrument captured in the image may be recognized by applying imageanalysis to the image.

Here, with reference to FIG. 22, the example of the control related tothe correction of image blurring according to the state of the surgicalinstrument will be described in further detail. FIG. 22 is a flowchartillustrating an example of a flow of control of the blurring correctionaccording to the state of the surgical instrument in the presentexample.

As illustrated in FIG. 22, the control apparatus 200 i (determiningsection 250 i) determines whether or not a predetermined surgicalinstrument has been detected (S201), and in a case where the surgicalinstrument has been detected (S201, YES), determines whether or not apredetermined operation with the surgical instrument has been detected(S203). Then, in a case where the operation with the surgical instrumenthas been detected (S203, YES), the control apparatus 200 i (secondcontrol section 217 i) provides control to increase the correctionintensity of the blurring correction (S205).

For example, FIG. 23 illustrates examples of the surgical instrument tobe detected and examples of the operation with the surgical instrument.Specifically, examples of the surgical instrument to be detected includean “stapler,” an “energy device,” a “suture needle and thread,” and a“debrider.” Additionally, examples of the operation with the staplerinclude an operation of “nipping” the living organism or the like, andan operation of “discharging” a drug or the like. Additionally, examplesof the operation with the energy device include an operation of“incising” the living organism or the like and a “hemostasis” operation.Additionally, an example of the operation with the suture needle andthread is a “suture” operation. Additionally, an example of theoperation with the debrider is an operation of “excising” lesion or thelike. In a case where these operations are performed, the imagedesirably remains stable without being blurred. Accordingly, when theoperation is detected, possible image blurring may be suppressed byproviding control to increase the correction intensity of the blurringcorrection.

On the other hand, as illustrated in FIG. 22, in a case where thepredetermined surgical instrument fails to have been detected (S201, NO)or the predetermined operation with the predetermined surgicalinstrument fails to have been detected (S203, NO), the control apparatus200 i (second control section 217 i) may provide control to reduce thecorrection intensity of the blurring correction (S205). Additionally, atthis time, the control apparatus 200 i (second control section 217 i)may disable (suppress) the blurring correction.

With reference to FIGS. 21 to 23, Example 9 has been described, which isan example of the control using, as input information, the informationindicative of the state of the surgical instrument and related to thecorrection of image blurring according to the state of the surgicalinstrument.

Example 10: Control According to User Input

Now, with reference to FIGS. 24 and 25, Example 10 will be describedthat is an example of control using an user input as input informationand related to correction of image blurring according to the user input.FIG. 24 and FIG. 25 are descriptive diagrams illustrating an example ofcontrol related to correction of image blurring performed by a controlapparatus according to Example 10. Note that, in this description, thecontrol apparatus according to the present example may be referred to asa “control apparatus 200 j” so as to be distinguished from controlapparatuses according to the other examples.

For example, FIG. 24 illustrates an example of a functionalconfiguration of the control apparatus 200 j according to Example 10.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 j, an imageprocessing unit 210 j includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 j differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 j, and a second control section 217j of the image processing unit 210 j. Thus, for operations of thecontrol apparatus 200 j, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 j that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

In the present example, an example will be described in which thecorrection intensity of the blurring correction is controlled inaccordance with an instruction from the user based on an operation via apredetermined input section. That is, the determining section 250 jacquires a user input via the predetermined input section as inputinformation, and determines a coefficient for controlling the degree ofapplication of the blurring correction (that is, the correctionintensity) in accordance with the acquired input information (that is,the user input). That is, the second control section 217 h determines acorrection coefficient indicative of the correction intensity of theblurring correction on the basis of the result of the determination madeby the determining section 250 h according to the user input.

For example, FIG. 25 illustrates an example of an input interface 180provided in the camera head or the like. In the example illustrated inFIG. 25, buttons 181 a to 181 c are provided to specify the correctionintensity of the blurring correction, and a button 181 d is provided tospecify disabling of the blurring correction. Additionally, in theexample illustrated in FIG. 25, to enable the blurring correction, anyof the buttons 181 a to 181 c can be operated to selectively switch thecorrection intensity of the blurring correction among three stages“high,” “medium,” and “low.”

With reference to FIG. 24 and FIG. 25, the example of the control hasbeen described that uses the user input as input information and that isrelated to the correction of image blurring in accordance with the userinput.

Example 11: Control According to CCU/Switching of Light Source Mode

Now, with reference to FIG. 26, Example 11 will be described that is anexample of control using, as input information, information indicativeof the CCU/switching of the light source mode and related to correctionof image blurring according to the switching. FIG. 26 is a descriptivediagram illustrating an example of control related to correction ofimage blurring performed by a control apparatus according to Example 11.Note that, in this description, the control apparatus according to thepresent example may be referred to as a “control apparatus 200 k” so asto be distinguished from control apparatuses according to the otherexamples.

For example, FIG. 26 illustrates an example of a functionalconfiguration of the control apparatus 200 k according to Example 11.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 k, an imageprocessing unit 210 k includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 k differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 k, and a second control section 217k of the image processing unit 210 k. Thus, for operations of thecontrol apparatus 200 k, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 k that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

For example, temporary vibration may be caused by an operation on theCCU (for example, changing the zoom magnification), an operation ofswitching the light source mode, or the like, resulting in temporaryimage blurring. When the blurring correction is applied even in such acase, the observation target such as the living organism may becomedifficult to observe. In light of such circumstances, in the presentexample, in a case where a predetermined operation such as the operationon the CCU or the operation of switching the light source mode isdetected, the control apparatus 200 k provides control to temporarilyreduce the correction intensity of the blurring correction ortemporarily disables the blurring correction.

That is, the determining section 250 k acquires, as input information,the information indicative of a detection result for the operation onthe CCU, the operation of switching the light source mode, or the like,and determines a coefficient for controlling the degree of applicationof the blurring correction (that is, the correction intensity) inaccordance with the acquired input information (that is, the informationindicative of the detection result for the operation on the CCU, theoperation of switching the light source mode, or the like). That is, thesecond control section 217 h determines a correction coefficientindicative of the correction intensity of the blurring correction on thebasis of the result of the determination made by the determining section250 h according to the detection result for the operation on the CCU,the operation of switching the light source mode, or the like.

Note that, in a case of having provided control to temporarily reducethe correction intensity of the blurring correction or havingtemporarily disabled the blurring correction, the control apparatus 200k may return, after elapse of a given time, the correction intensity ofthe blurring correction to a state before the control.

With reference to FIG. 26, Example 11 has been described, which is anexample of the control using, as input information, the informationindicative of the CCU/switching of the light source mode and related tothe correction of image blurring according to the switching.

Example 12: Control According to Calculation Result for Motion Vectors

Now, with reference to FIG. 27, Example 12 will be described that is anexample of control using, as input information, information indicativeof a calculation result for motion vectors and related to correction ofimage blurring according to the calculation result for the motionvectors. FIG. 27 is a descriptive diagram illustrating an example ofcontrol related to correction of image blurring performed by a controlapparatus according to Example 12. Note that, in this description, thecontrol apparatus according to the present example may be referred to asa “control apparatus 200 m” so as to be distinguished from controlapparatuses according to the other examples.

For example, FIG. 27 illustrates an example of a functionalconfiguration of the control apparatus 200 m according to Example 12.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 m, an imageprocessing unit 210 m includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 m differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 m, and a second control section 217m of the image processing unit 210 m. Thus, for operations of thecontrol apparatus 200 m, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 m that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

The determining section 250 m acquires, as input information, theinformation indicative of the result of estimation of motion vectorsperformed by the estimating section 215, and determines a coefficientfor controlling the degree of application of the blurring correction(that is, the correction intensity) in accordance with the acquiredinput information (that is, the information indicative of the estimationresult for the motion vectors). That is, the second control section 217m determines a correction coefficient indicative of the correctionintensity of the blurring correction on the basis of the result of thedetermination made by the determining section 250 m according to theestimation result for the motion vectors.

By way of a specific example, in a case where a predetermined operationis performed such as an operation of translating the imaging section asin insertion of the endoscope or what is called a pan operation, theposition or orientation of the imaging section is changed in conjunctionwith the operation, and the change is detected as a motion vector. In acase where such an intentional operation is performed, when the blurringcorrection is applied even to motion of the image involved in theoperation, the observation target such as the living organism may becomedifficult to observe. Thus, for example, in a case where the estimationof the motion vectors is recognized to result from a change in theposition or orientation of the imaging section by the intentionaloperation, the observation target such as the living organism may bemade easier to observe by providing control to reduce the correctionintensity or temporarily disabling the blurring correction.

Note that a method for recognizing that the estimation of the motionvectors results from a change in the position or orientation of theimaging section by an intentional operation is not particularly limitedas long as the recognition can be achieved. For example, when theendoscope is inserted or in a case where a pan operation is performed,the entire image may change steadily in a given direction. In such acase, for example, the estimation result for the motion vectors tends toindicate a substantially equal direction among a plurality of frames andthe magnitude of each motion vector tends to vary slowly. Suchproperties can be utilized in the following manner: for example, in acase where a change in the entire image is recognized to be a steadychange in a given direction instead of an instantaneous change, theposition or orientation of the imaging section can be recognized to havebeen changed by an intentional operation as in the insertion of theendoscope, the pan operation, or the like.

With reference to FIG. 27, Example 12 has been described, which is anexample of the control using, as input information, the informationindicative of the calculation result for the motion vectors and relatedto the correction of image blurring according to the calculation resultfor the motion vectors.

Example 13: Control According to Detection Status of AF/AE

Now, with reference to FIG. 28, Example 13 will be described that is anexample of control using, as input information, information indicativeof a detection status of AF (Autofocus)/AE (Automatic Exposure) andrelated to correction of image blurring according to the detectionstatus of the AF/AE. FIG. 28 is a descriptive diagram illustrating anexample of control related to correction of image blurring performed bya control apparatus according to Example 13. Note that, in thisdescription, the control apparatus according to the present example maybe referred to as a “control apparatus 200 n” so as to be distinguishedfrom control apparatuses according to the other examples.

For example, FIG. 28 illustrates an example of a functionalconfiguration of the control apparatus 200 n according to Example 13.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 n, an imageprocessing unit 210 n includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 n differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 n, and a second control section 217n of the image processing unit 210 n. Thus, for operations of thecontrol apparatus 200 n, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 n that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

The determining section 250 n acquires, as input information, theinformation indicative of the detection status of AF or AE, anddetermines a coefficient for controlling the degree of application ofthe blurring correction (that is, the correction intensity) inaccordance with the acquired input information (that is, the informationindicative of the detection status of AF or AE). That is, the secondcontrol section 217 n determines a correction coefficient indicative ofthe correction intensity of the blurring correction on the basis of theresult of the determination made by the determining section 250 naccording to the detection status of AF or AE. Note that a method forrecognizing the detection status of AF or AE is not particularly limitedas long as the recognition can be achieved. By way of example of anotherexample, information indicating the detection status of AF or AE from animage sensor or the like provided in the imaging section may be acquiredfrom the imaging section (or the image sensor itself). Additionally, byway of another example, the detection status of AF or AE may beestimated by analyzing the captured image.

Specifically, a failure in the detection of AF or AE may prevent correctextraction of feature points or correct estimation of motion vectorsbased on the result of extraction of the feature points, resulting inreduced reliability of the correction value for the blurring correction.By way of a more specific example, a failure in AF leads to blurring ofthe image, making extraction of feature points such as edges difficult.Additionally, in a case where a failure in AF results in reducedlightness of the entire image, extracting feature points is difficult.

It is assumed that, in a case where the correction intensity of theblurring correction is increased with the reliability of the correctionvalue for the blurring correction reduced as described above, theblurring correction fails to be applied in an appropriate manner, makingthe observation target such as the living organism difficult to observe.Thus, for example, in a case where the detection of AF or Ae fails, theobservation target such as the living organism can be prevented frombecoming difficult to observe by reducing the correction intensity ofthe blurring correction or temporarily disabling the blurringcorrection.

With reference to FIG. 28, Example 13 has been described, which is anexample of the control using, as input information, the informationindicative of the detection status of AF (Autofocus)/AE (AutomaticExposure) and related to the correction of image blurring according tothe detection status of AF/AE.

Example 14: Control According to Detection Result for Lens Stain

Now, with reference to FIG. 29, Example 14 will be described that is anexample of control using, as input information, information indicativeof a detection result for lens stain and related to correction of imageblurring according to the detection result for lens stain. FIG. 29 is adescriptive diagram illustrating an example of control related tocorrection of image blurring performed by a control apparatus accordingto Example 14. Note that, in this description, the control apparatusaccording to the present example may be referred to as a “controlapparatus 200 p” so as to be distinguished from control apparatusesaccording to the other examples.

For example, FIG. 29 illustrates an example of a functionalconfiguration of the control apparatus 200 p according to Example 14.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 p, an imageprocessing unit 210 p includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 p differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 p, and a second control section 217p of the image processing unit 210 p. Thus, for operations of thecontrol apparatus 200 p, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 p that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

The determining section 250 p acquires, as input information, theinformation indicative of the detection result for lens stain, anddetermines a coefficient for controlling the degree of application ofthe blurring correction (that is, the correction intensity) inaccordance with the acquired input information (that is, the informationindicative of the detection result for lens stain). That is, the secondcontrol section 217 p determines a correction coefficient indicative ofthe correction intensity of the blurring correction on the basis of theresult of the determination made by the determining section 250 paccording to the detection result for lens stain. Note that a method fordetecting lens stain is not particularly limited as long as the methodallows lens stain to be detected. For example, lens stain may bedetected by various sensors or the like. Additionally, by way of anotherexample, lens stain may be detected by analyzing the captured image.

Specifically, under circumstances where the lens is stained, the stainmay shield a subject corresponding to the observation target such as theliving organism or blur the image. Under such circumstances, featurepoints may be prevented from being correctly extracted or motion vectorsmay be prevented from being correctly estimated on the basis of theresult of extraction of the feature points, resulting in reducedreliability of the correction value for the blurring correction.

It is assumed that, in a case where the correction intensity of theblurring correction is increased with the reliability of the correctionvalue for the blurring correction reduced as described above, theblurring correction fails to be applied in an appropriate manner, makingthe observation target such as the living organism difficult to observe.Thus, for example, in a case where stain attached to the lens isdetected, the observation target such as the living organism can beprevented from becoming difficult to observe by reducing the correctionintensity of the blurring correction or temporarily disabling theblurring correction.

With reference to FIG. 29, Example 14 has been described, which is anexample of the control using, as input information, the informationindicative of the detection result for lens stain and related to thecorrection of image blurring according to the detection result for lensstain.

Example 15: Control According to Detection Result for Smoke or Mist

Now, with reference to FIG. 30, Example 15 will be described that is anexample of control using, as input information, information indicativeof a detection result for smoke or mist and related to correction ofimage blurring according to the detection result for smoke or mist. FIG.30 is a descriptive diagram illustrating an example of control relatedto correction of image blurring performed by a control apparatusaccording to Example 15. Note that, in this description, the controlapparatus according to the present example may be referred to as a“control apparatus 200 q” so as to be distinguished from controlapparatuses according to the other examples.

For example, FIG. 30 illustrates an example of a functionalconfiguration of the control apparatus 200 q according to Example 15.Note that a precondition for the present example is that image blurringis to be corrected. Thus, in the control apparatus 200 q, an imageprocessing unit 210 q includes no component corresponding to the firstcontrol section 211 described above with reference to FIG. 3.Additionally, the control apparatus 200 q differs from the controlapparatus 200 described with reference to FIG. 3 mainly in the operationof each of a determining section 250 q, and a second control section 217q of the image processing unit 210 q. Thus, for operations of thecontrol apparatus 200 q, FIG. 3 will be described with focus on sectionsdifferent from the corresponding sections of the control apparatus 200.Sections of the control apparatus 200 q that are substantially similarto the corresponding sections of the control apparatus 200 will not bedescribed below in detail.

Specifically, it is assumed that mist caused by spraying of a drug orthe like temporarily shields a subject corresponding to the observationtarget such as the living organism. It is also assumed that smoke causedby the use of an energy device or the like temporarily shields a subjectcorresponding to the observation target such as the living organism.Under circumstances where a visible gaseous substance such as smoke ormist thus temporarily shields a subject corresponding to the observationtarget such as the living organism, feature points may be prevented frombeing correctly extracted or motion vectors may be prevented from beingcorrectly estimated on the basis of the result of extraction of thefeature points, resulting in reduced reliability of the correction valuefor the blurring correction. It is assumed that, in a case where thecorrection intensity of the blurring correction is increased with thereliability of the correction value for the blurring correction reducedas described above, the blurring correction fails to be applied in anappropriate manner, making the observation target such as the livingorganism difficult to observe.

The determining section 250 q may acquire, as input information, theinformation indicative of the detection result for smoke, mist, or thelike, and determine a coefficient for controlling the degree ofapplication of the blurring correction (that is, the correctionintensity) in accordance with the acquired input information (that is,the information indicative of the detection result for the gaseoussubstance). That is, the second control section 217 p determines acorrection coefficient indicative of the correction intensity of theblurring correction on the basis of the result of the determination madeby the determining section 250 p according to the detection result for avisible gaseous substance such as smoke or mist. Note that a method fordetecting a visible gaseous substance such as smoke or mist is notparticularly limited as long as the method allows the visible gaseoussubstance to be detected. For example, the gaseous substance may bedetected by various sensors or the like. Additionally, by way of anotherexample, the gaseous substance may be detected by analyzing the capturedimage.

Thus, for example, in a case where the visible gaseous substance such assmoke or mist is detected, the observation target such as the livingorganism can be prevented from becoming difficult to observe by reducingthe correction intensity of the blurring correction or temporarilydisabling the blurring correction. Note that, on the basis of a conceptsimilar to the concept of Example 5 described above, the correctionintensity of the blurring correction may be controlled according to theratio of a part of the subject corresponding to the observation targetsuch as the living organism which part is shielded, in the screen, bythe visible gaseous substance such as smoke or mist.

With reference to FIG. 30, Example 15 has been described, which is anexample of the control using, as input information, the informationindicative of the detection result for smoke or mist and related to thecorrection of image blurring according to the detection result for smokeor mist.

Example 16: Example of Control According to Procedure

Now, Example 16 will be described that is an example in which each ofthe above-described examples is appropriately applied according to aprocedure, a correction target, a technique, or the like in surgery orthe like.

By way of a specific example, an operation in gastrointestinal surgeryis assumed, and procedures are assumed to be executed in the order of“incision and exfoliation,” “biopsy,” and “blood vessel treatment.” Insuch a case, for example, in the “incision and exfoliation” procedure,whether or not to apply the blurring correction or the correctionintensity of the blurring correction may be controlled according to atleast any one of the types of input information such as the occupancy ofa surgical instrument in the screen, the state of the surgicalinstrument, or the detection result for smoke or mist. Additionally, inthe “biopsy” procedure, whether or not to apply the blurring correctionor the correction intensity of the blurring correction may be controlledaccording to at least any one of the types of input information such asthe occupancy of a surgical instrument in the screen and the state ofthe surgical instrument. Additionally, in the “blood vessel treatment”procedure, whether or not to apply the blurring correction or thecorrection intensity of the blurring correction may be controlledaccording to at least any one of the types of input information such asthe zoom magnification, the working distance, and the detection resultfor lens stain.

Additionally, for lengthy surgery, whether or not to apply the blurringcorrection or the correction intensity of the blurring correction may becontrolled, for example, according to the operative duration.Additionally, under circumstances where the operator moves, whether ornot to apply the blurring correction or the correction intensity of theblurring correction may be controlled, for example, according to thedistance between the operator and the monitor.

Note that the coefficient for controlling the correction intensity ofthe blurring correction may be calculated in accordance with a pluralityof types of input information. In such a case, for example, thecoefficients corresponding to the respective types of input informationmay be multiplied together to calculate a coefficient to be finallyapplied. Additionally, the coefficients corresponding to the respectivetypes of input information may be weighted according to a utilizationscene.

Example 16 has been described that is an example in which each of theabove-described examples is appropriately applied according to theprocedure, the correction target, the technique, or the like in thesurgery or the like.

3. Applied Example

Now, as an applied example of a medical observation system according tothe embodiment of the present disclosure, an example of the medicalobservation system configured as a microscopic imaging system includinga microscope unit will be described with reference to FIG. 31.

FIG. 31 is a descriptive diagram illustrating the applied example of themedical observation system according to the embodiment of the presentdisclosure and also illustrating an example of a general configurationof the microscopic imaging system. Specifically, FIG. 31 illustrates anexample of the use of a surgical video microscope apparatus including anarm, as an applied example of the use of the medical observation systemaccording to the embodiment of the present disclosure.

For example, FIG. 31 schematically illustrates the manner of medicaltreatment using the surgical video microscope apparatus. Specifically,FIG. 31 illustrates that a surgeon who is an operator (user) 820 isoperating on a medical treatment target (patient) 840 on an operatingtable 830 using a surgical instrument 821, for example, a scalpel,tweezers, or forceps. In the following description, the medicaltreatment is intended to be a general term for various medicaltreatments such as surgery and examinations executed on the patient, whois the medical treatment target 840, by the surgeon, who is the user820. Additionally, in the example illustrated in FIG. 31, the manner ofsurgery is illustrated as an example of medical treatment. However, themedical treatment using the surgical video microscope apparatus 810 isnot limited to surgery but may be any of various other medicaltreatments.

The surgical video microscope apparatus 810 is provided beside theoperating table 830. The surgical video microscope apparatus 810includes a base section 811 corresponding to a base, an arm section 812extending from the base section 811, and an imaging unit 815 connectedto a distal end of the arm section 812 as a distal-end unit. The armsection 812 includes a plurality of joint sections 813 a, 813 b, and 813c, a plurality of links 814 a and 814 b coupled together by the jointsections 813 a and 813 b, and an imaging unit 815 provided at a distalend of the arm section 812. In the example illustrated in FIG. 31, thearm section 812 includes the three joint sections 813 a to 813 c and thetwo links 814 a and 814 b for simplification. However, in actuality, thenumbers and shapes of the joint sections 813 a to 813 c and the links814 a and 814 b, the directions of driving shafts of the joint sections813 a to 813 c, and the like may be appropriately set so as to achieve adesired degree of freedom with the degrees of freedom of positions andpostures of the arm section 812 and the imaging unit 815 taken intoaccount.

The joint sections 813 a to 813 c have a function to pivotally couplethe links 814 a and 814 b together, and are rotationally driven tocontrol driving of the arm section 812. Here, in the description below,the position of each component of the surgical video microscopeapparatus 810 means a position (coordinates) in a space defined fordriving control. The posture of each component means the orientation(angle) with respect to an optional axis in the space defined fordriving control. Additionally, in the description below, driving (ordriving control) of the arm section 812 refers to a change in theposition and orientation of each component of the arm section 812effected by driving (or driving control) of the joint sections 813 a to813 c and driving (or driving control) of the joint sections 813 a to813 c (or refers to the control of the change in the position andposture).

The imaging unit 815 is connected to a distal end of the arm section 812as a distal-end unit. The imaging unit 815 is a unit that acquires animage of an imaging target, for example, a camera capable of capturingmoving images or still images. As illustrated in FIG. 31, to allow theimaging unit 815 provided at the distal end of the arm section 812 toimage a medical treatment site of the medical treatment target 840, thesurgical video microscope apparatus 810 controls the postures andpositions of the arm section 812 and the imaging unit 815. Note that theconfiguration of the imaging unit 815 connected to the distal end of thearm section 812 as a distal-end unit is not particularly limited andthat, for example, the imaging unit 815 is configured as a microscopethat acquires an enlarged image of the imaging target. Additionally, theimaging unit 815 may be configured such that the imaging unit 815 can beinstalled on and removed from the arm section 812. In such aconfiguration, for example, the imaging unit 815 corresponding to anintended use may be appropriately connected to the distal end of the armsection 812 as a distal-end unit. Note that, as the imaging unit 815,for example, an imaging apparatus can be applied to which a branchingoptical system according to the above-described embodiment is applied.Additionally, this description focuses on the application of the imagingunit 815 as a distal-end unit. However, the distal-end unit connected tothe distal end of the arm section 812 is not necessarily limited to theimaging unit 815.

Additionally, a display apparatus 850 such as a monitor or a display isinstalled at a position opposite to the user 820. An image of themedical treatment site captured by the imaging unit 815 is displayed ona display screen of the display apparatus 850 as an electronic image.The user 820 executes various treatments while viewing the electronicimage of the medical treatment site displayed on the display screen ofthe display apparatus 850.

The configuration described above allows surgery to be performed withthe medical treatment site being imaged by the surgical video microscopeapparatus 810.

4. Example of Hardware Configuration

Now, with reference to FIG. 32, an example of a hardware configurationof what is called an information processing apparatus executing varioustypes of processing, such as the CCU in the above-described endoscopicsurgery system. FIG. 32 is a functional block diagram illustrating aconfiguration example of a hardware configuration of an informationprocessing apparatus constituting a medical observation system accordingto the embodiment of the present disclosure.

An information processing apparatus 900 constituting the medicalobservation system according to the present embodiment mainly includes aCPU 901, a ROM 902, and a RAM 903. Additionally, the informationprocessing apparatus 900 further includes a host bus 907, a bridge 909,an external bus 911, an interface 913, an input apparatus 915, an outputapparatus 917, a storage apparatus 919, a drive 921, a connection port923, and a communication apparatus 925.

The CPU 901 functions as an arithmetic processing apparatus and acontrol apparatus to control operations in general in the informationprocessing apparatus 900 or some of the operations in accordance withvarious programs recorded in the ROM 902, the RAM 903, the storageapparatus 919, or a removable recording medium 927. The ROM 902 storesprograms, arithmetic parameters, and the like used by the CPU 901. TheRAM 903 primarily stores programs used by the CPU 901, and for example,parameters appropriately varying during execution of the programs. Theabove-described components are connected together by the host bus 907including an internal bus such as a CPU bus. For example, the imageprocessing unit 210 and the determining section 250 described withreference to FIG. 3 may include the CPU 901.

The host bus 907 is connected to the external bus 911 such as a PCI(Peripheral Component Interconnect/Interface) bus via the bridge 909.Additionally, the external bus 911 connects to the input apparatus 915,the output apparatus 917, the storage apparatus 919, the drive 921, theconnection port 923, and the communication apparatus 925 via theinterface 913.

The input apparatus 915 is operating means operated by the user, such asa mouse, a keyboard, a touch panel, a button, a switch, a lever, and apedal. The input apparatus 915 may also be remote control means (what iscalled a remote controller) utilizing infrared rays or other radiowaves, or external connection equipment 929 such as a cellular phone ora PDA supporting operation of the information processing apparatus 900.The input apparatus 915 includes, for example, an input control circuitthat generates an input signal based on information input by the userusing the above-described operating means and that outputs the inputsignal to the CPU 901. By operating the input apparatus 915, the user ofthe information processing apparatus 900 can input various data to theinformation processing apparatus 900 and provide the informationprocessing apparatus 900 with instructions to perform processingoperations.

The output apparatus 917 includes an apparatus capable of visually orauditorily notifying the user of acquired information. Examples of suchan apparatus include display apparatuses such as a CRT displayapparatus, a liquid crystal display apparatus, a plasma displayapparatus, an EL display apparatus, and a lamp, sound output apparatusessuch as a speaker and a headphone, and a printer apparatus. The outputapparatus 917 outputs results obtained from various types of processingexecuted by the information processing apparatus 900. Specifically, thedisplay apparatus displays, in text or as images, results obtained fromvarious types of processing executed by the information processingapparatus 900. On the other hand, the sound output apparatus converts,into an analog signal, an audio signal consisting of reproduced sounddata, acoustic data, or the like, and outputs the analog signal.

The storage apparatus 919 is a apparatus for data storage configured asa part of a storage section of the information processing apparatus 900.The storage apparatus 919 includes, for example, a magnetic storagedevice such as an HDD (Hard Disk Drive), a semiconductor storage device,or an optical storage device or a photomagnetic storage device. Thestorage apparatus 919 stores programs executed by the CPU 901, variousdata, and the like.

The drive 921 is a reader/writer for recording media built in theinformation processing apparatus 900 or an external reader/writer forrecording media attached to the information processing apparatus 900.The drive 921 reads information recorded in the removable recordingmedium 927 such as a magnetic disk, an optical disc, a photomagneticdisk, or a semiconductor memory that is mounted in the drive 921, andoutputs the information to the RAM 903. Additionally, the drive 921 canwrite recording into the removable recording medium 927 such as amagnetic disk, an optical disc, a photomagnetic disk, or a semiconductormemory that is mounted in the drive 921. The removable recording medium927 is, for example, a DVD medium, an HD-DVD medium, or a Blu-ray(registered trademark) medium. Additionally, the removable recordingmedium 927 may be a CompactFlash (registered trademark) (CF), a flashmemory, an SD memory card (Secure Digital memory card), or the like.Additionally, the removable recording medium 927 may be, for example, anIC card (Integrated Circuit card) or electronic instrument in which anon-contact IC chip is mounted.

The connection port 923 is a port for direct connection to theinformation processing apparatus 900. An example of the connection port923 is a USB (Universal Serial Bus) port, an IEEE 1394 port, or an SCSI(Small Computer System Interface) port. Another example of theconnection port 923 is an RS-232C port, an optical audio terminal, anHDMI (registered trademark) (High-Definition Multimedia Interface) port,or the like. Connecting the external connection equipment 929 to theconnection port 923 allows the information processing apparatus 900 toacquire various data directly from the external connection equipment 929and provide various data to the external connection equipment 929.

The communication apparatus 925 is a communication interface including,for example, a communication device to be connected to a communicationnetwork (network) 931. The communication apparatus 925 is, for example,a communication card for a wired or wireless LAN (Local Area Network),Bluetooth (registered trademark), or WUSB (Wireless USB). Additionally,the communication apparatus 925 may be a router for opticalcommunication, a router for ADSL (Asymmetric Digital Subscriber Line), amodem for various communications, or the like. The communicationapparatus 925 can transmit and receive signals and the like, forexample, to and from the Internet or any other communication equipmentin accordance with a predetermined protocol, for example, TCP/IP.Additionally, the communication network 931 connected to thecommunication apparatus 925 includes a network connected by wire orwirelessly to the communication apparatus 925, and may be, for example,the Internet, a home LAN, infrared communication, radio wavecommunication, or satellite communication.

The example of the hardware configuration has been disclosed that canimplement the functions of the information processing apparatus 900constituting the medical observation system according to the embodimentof the present disclosure. Each of the above-described components mayinclude a general-purpose member or hardware dedicated to the functionsof the component. Accordingly, the hardware configuration can beappropriately varied according to the technique level of the moment whenthe present embodiment is implemented. Note that, although notillustrated in FIG. 32, the configuration, needless to say, includesvarious components corresponding to the information processing apparatus900 constituting the medical observation system.

Note that a computer program can be created that is configured torealize each function of the information processing apparatus 900constituting the medical observation system according to the presentembodiment described above and that the computer program can beimplemented in a personal computer or the like. Additionally, acomputer-readable recording medium can be provided in which such acomputer program is stored. The recording medium is, for example, amagnetic disk, an optical disc, a photomagnetic disk, or a flash memory.Additionally, the above-described computer program may be distributed,for example, via the network instead of using the recording medium.Additionally, the number of computers caused to execute the computerprogram is not particularly limited. For example, the computer programmay be executed by a plurality of computers (for example, a plurality ofservers) in cooperation.

5. Conclusion

As described above, in the medical observation system according to thepresent embodiment, the control apparatus 200 controls whether or not toapply correction of image blurring and the intensity of the correctionin a case where the correction is applied, on the basis of the inputinformation corresponding to various states or circumstances.

By way of a specific example, the control apparatus 200 may providecontrol to increase the degree of correction of image blurringconsistently with zoom magnification of the imaging section such as thecamera head. In other words, the control apparatus 200 may providecontrol to reduce the degree of correction of image blurringconsistently with zoom magnification. With such a configuration, evenunder circumstances where blurring correction is performed on the basisof motion vectors indicative of motion of a surgical instrument such asintensely shaking forceps and where a bird's-eye view imagecorresponding to the background shakes, the correction intensity of theblurring correction is reduced to suppress shaking of the bird's eyeview, making the observation target such as the living organism easierto observe. Additionally, even in a case where the range imaged by theimaging section is intentionally moved such as a case where the scope ofthe endoscope apparatus or the like is intentionally moved, thecorrection intensity of the blurring correction is reduced to suppresscorrection of image blurring caused by the intentional motion, makingthe observation target such as the living organism easier to observe.

Additionally, the control apparatus 200 may control the operationsrelated to the correction of image blurring according to the ratio of aregion of the subject corresponding to the observation target such asthe living organism whose region is shielded, in the screen (in otherwords, the input image signal), by a subject such as the surgicalinstrument or the visible gaseous substance that is different from theobservation target. By way of a more specific example, the controlapparatus 200 may reduce the degree of correction of image blurring withincreasing ratio of the region of the subject corresponding to theobservation target whose region is shielded by another subject, ordisable application of image blurring. With such a configuration, forexample, even under circumstances where an increased occupancy of theanother subject in the screen makes extraction of feature amountsdifficult to reduce the reliability of the correction value for theblurring correction, the observation target such as the living organismcan be prevented from becoming difficult to observe by reducing thecorrection intensity of the blurring correction.

Needless to say, the above-described configuration is only an exampleand is not intended to limit the contents of the control related to thecorrection of image blurring performed by the control apparatus 200according to the present embodiment. As input information, informationrelated to an imaging environment for the observation target such as theliving organism can be utilized, the information being, for example, thestate of the surgical instrument or motion (vibration) of the bed.Additionally, by way of another example, instead of the zoommagnification, information related to the imaging conditions for theobservation target such as the living organism, for example, informationrelated to AF/AE, can be utilized as input information. Additionally, onthe basis of the property that image blurring is estimated by extractingfeature amounts from the input image signal, a detection result for afactor making extraction of the feature amounts difficult may beutilized as input information. Note that, in this case, the observationtarget such as the living organism can be prevented from becomingdifficult to observe by, for example, reducing the correction intensityof correction of image blurring or temporarily disabling the correctionaccording to the detection result for the factor making extraction ofthe feature amounts difficult.

The preferred embodiment of the present disclosure have been describedin detail with reference to the attached drawings. However, thetechnical scope of the present disclosure is not limited to suchexamples. A person with an ordinary skill in the art of the presentdisclosure may obviously arrive at many varied or modified exampleswithin the range of technical concepts recited in claims, and it isunderstood that these examples, needless to say, belong to the technicalscope of the present disclosure.

Additionally, the effects described herein are only descriptive orillustrative and not restrictive. In other words, in addition to orinstead of the above-described effects, the technique according to thepresent disclosure may produce other effects obvious, from thedescription herein, to a person with an ordinary skill in the art.

Note that the following configurations also belong to the technicalscope of the present disclosure.

(1) A control apparatus including:

an estimating section calculating motion of an entire image on the basisof an image signal corresponding to an optical image of a livingorganism input from a predetermined imaging section, to estimateblurring of the entire image according to a result of the calculation;and

a control section controlling an operation related to correction of theblurring of the entire image by controlling a coefficient forcontrolling an amount of correction of the blurring on the basis of azoom magnification of the imaging section such that a degree ofcorrection of the blurring increases consistently with the zoommagnification.

(2) The control apparatus according to (1) described above, in which

the control section provides control to suppress correction of theblurring in a case where the zoom magnification is lower than or equalto a threshold.

(3) The control apparatus according to (1) or (2) described above,including:

an extracting section extracting a feature point from the image signal,in which

the estimating section calculates the motion of the entire image on thebasis of a result of extraction of the feature point.

(4) A control apparatus including:

an estimating section calculating motion of an entire image on the basisof an image signal corresponding to an optical image of a livingorganism input from a predetermined imaging section, to estimateblurring of the entire image according to a result of the calculation,and

a control section controlling an operation related to correction of theblurring of the entire image on the basis of a ratio of a region of theliving organism in the image signal, the region being shielded by asubject different from the living organism.

(5) The control apparatus according to (4) described above, in which

the control section controls a coefficient for controlling the amount ofcorrection of the blurring according to the ratio of the region.

(6) The control apparatus according to (5) described above, in which

the control section controls the coefficient to reduce a degree ofcorrection of the blurring with an increasing ratio of the region.

(7) The control apparatus according to (4) described above, in which

the control section provides control to suppress correction of theblurring according to the ratio of the region.

(8) The control apparatus according to any one of (4) to (7) describedabove, in which

the subject includes a surgical instrument.

(9) The control apparatus according to any one of (4) to (7) describedabove, in which

the subject includes a visible gaseous substance.

(10) The control apparatus according to any one of (4) to (9) describedabove, in which

the ratio of the region is calculated according to a recognition resultfor the subject.

(11) The control apparatus according to any one of (4) to (10) describedabove, including:

an extracting section extracting a feature point from the image signal,in which

the estimating section calculates the motion of the entire image on thebasis of a result of extraction of the feature point.

(12) A control method including:

calculating, by a computer, motion of an entire image on the basis of animage signal corresponding to an optical image of a living organisminput from a predetermined imaging section, to estimate blurring of theentire image according to a result of the calculation; and

controlling, by the computer, an operation related to correction of theblurring of the entire image by controlling a coefficient forcontrolling an amount of correction of the blurring on the basis of azoom magnification of the imaging section such that a degree ofcorrection of the blurring increases consistently with the zoommagnification.

(13) A control method including:

calculating, by a computer, motion of an entire image on the basis of animage signal corresponding to an optical image of a living organisminput from a predetermined imaging section, to estimate blurring of theentire image according to a result of the calculation, and

controlling, by the computer, an operation related to correction of theblurring of the entire image on the basis of a ratio of a region of theliving organism in the image signal, the region being shielded by asubject different from the living organism.

(14) A program causing a computer to execute:

calculating motion of an entire image on the basis of an image signalcorresponding to an optical image of a living organism input from apredetermined imaging section, to estimate blurring of the entire imageaccording to a result of the calculation; and

controlling an operation related to correction of the blurring of theentire image by controlling a coefficient for controlling an amount ofcorrection of the blurring on the basis of a zoom magnification of theimaging section such that a degree of correction of the blurringincreases consistently with the zoom magnification.

(15) A program causing a computer to execute:

calculating motion of an entire image on the basis of an image signalcorresponding to an optical image of a living organism input from animaging section of a medical observation apparatus, to estimate blurringof the entire image according to a result of the calculation; and

controlling an operation related to correction of the blurring of theentire image on the basis of a ratio of a region of the living organismin the image signal, the region being shielded by a subject differentfrom the living organism.

REFERENCE SIGNS LIST

-   -   100 Endoscopic surgery system    -   101 Endoscope    -   117 Surgical instrument    -   127 Support arm apparatus    -   139 Camera control unit    -   141 Display apparatus    -   143 Light source apparatus    -   145 Arm control apparatus    -   147 Input apparatus    -   149 Treatment instrument control apparatus    -   151 Insufflation apparatus    -   153 Recorder    -   155 Printer    -   200 Control apparatus    -   210 Image processing unit    -   211 First control section    -   215 Estimating section    -   217 Second control section    -   219 Correction processing section    -   250 Determining section

The invention claimed is:
 1. A control apparatus, comprising: circuitryconfigured to: calculate motion of an entire image based on an imagesignal corresponding to an optical image of a living organism, whereinthe image signal is received from an imaging section of a medicalobservation apparatus; estimate blurring of the entire image based onthe calculation of the motion; determine an increase in occupancy of asubject different from the living organism in the image signal;determine an increase in a ratio of a region of the living organism thatis shielded by the subject in the image signal, wherein the increase inthe ratio is determined based on the increase in the occupancy of thesubject; and control an operation to reduce a degree of correction ofthe blurring of the entire image based on the determined increase in theratio of the region of the living organism shielded by the subject inthe image signal.
 2. The control apparatus according to claim 1, whereinthe circuitry is further configured to control a coefficient for thereduction of the degree of the correction of the blurring according tothe ratio of the region.
 3. The control apparatus according to claim 1,wherein the circuitry is further configured to suppress the correctionof the blurring according to the ratio of the region.
 4. The controlapparatus according to claim 1, wherein the subject includes a surgicalinstrument.
 5. The control apparatus according to claim 1, wherein thesubject that shields the region of the living organism includes avisible gaseous substance.
 6. The control apparatus according to claim1, wherein the circuitry is further configured to calculate the ratio ofthe region based on a recognition result of the subject.
 7. The controlapparatus according to claim 1, wherein the circuitry is furtherconfigured to: extract a feature point from the image signal; andcalculate the motion of the entire image based on the extraction of thefeature point.
 8. A control method, comprising: calculating, by acomputer, motion of an entire image based on an image signalcorresponding to an optical image of a living organism, wherein theimage signal is received from an imaging section of a medicalobservation apparatus; estimating, by the computer, blurring of theentire image based on the calculation of the motion; determining, by thecomputer, an increase in occupancy of a subject different from theliving organism in the image signal; determining, by the computer, anincrease in a ratio of a region of the living organism that is shieldedby the subject in the image signal, wherein the increase in the ratio isdetermined based on the increase in the occupancy of the subject; andcontrolling, by the computer, an operation to reduce a degree ofcorrection of the blurring of the entire image based on the determinedincrease in the ratio of the region of the living organism shielded bythe subject in the image signal.
 9. A non-transitory computer-readablemedium having stored thereon, computer-executable instructions which,when executed by a computer, cause the computer to execute operations,the operations comprising: calculating motion of an entire image basedon an image signal corresponding to an optical image of a livingorganism, wherein the image signal is received from an imaging sectionof a medical observation apparatus; estimating blurring of the entireimage based on the calculation of the motion; determining an increase inoccupancy of a subject different from the living organism in the imagesignal; determining an increase in a ratio of a region of the livingorganism that is shielded by the subject in the image signal, whereinthe increase in the ratio is determined based on the increase in theoccupancy of the subject; and controlling an operation to reduce adegree of correction of the blurring of the entire image based on thedetermined increase in the ratio of the region of the living organismshielded by the subject in the image signal.